RU2799572C1 - Method for signaling a set of output layers with a subimage - Google Patents

Abstract

FIELD: video encoding.

SUBSTANCE: video data having a plurality of layers is received. One or more first syntax elements are identified that indicate one or more sets of output layers corresponding to output layers from a plurality of layers of received video data. One or more output layers corresponding to the specified sets of output layers are decoded and displayed, wherein one or more syntax elements are signalled in the high-level syntax structure. Moreover, the high-level syntax structure comprises one of the following: a set of video parameters, a set of dependency parameters, a set of sequence parameters, a set of image parameters, a set of adaptation parameters, and reporting additional enhancement information. Here, the sequence parameter set is available for decoding before the sequence parameter set is referenced by one or more of the image parameter sets.

EFFECT: increased efficiency of video coding.

16 cl, 21 dwg

Description

CROSS-REFERENCE TO RELATED APPLICATION

[1] This application claims priority of U.S. Provisional Application No. 62/903,660, filed September 20, 2019, and U.S. Patent Application No. 17/021,243, filed September 15, 2020, which are incorporated herein in their entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[2] This disclosure relates generally to the field of video encoding and decoding, and more specifically to the reference and scope of a set of parameters in an encoded video stream.

BACKGROUND OF THE INVENTION

[3] Video encoding and decoding using motion-compensated inter-picture prediction has been known for decades. Uncompressed digital video may consist of a series of images, each image having a spatial size of, for example, 1920×1080 luma samples and associated chrominance samples. A series of images can have a fixed or variable image rate (informally also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 brightness sample resolution at 60Hz frame rate) requires bandwidth close to 1.5Gbps. An hour of such video requires more than 600 GB of disk space.

[4] One of the goals of video encoding and decoding may be to reduce the redundancy of the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, as well as a combination of both, can be used. Lossless compression refers to methods in which an exact copy of the original signal can be restored from a compressed source signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal suitable for the intended application. In the case of video, lossy compression is widely used. The amount of distortion allowed depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of broadcast television applications. The achievable compression ratio may reflect the following: higher distortion resolution/tolerance may result in a higher compression ratio.

[5] The video encoder and video decoder may use techniques from several broad categories including, for example, motion compensation, transformation, quantization, and entropy coding, some of which will be presented below.

[6] Historically, video encoders and video decoders tended to work with a given picture size, which in most cases was defined and held constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2, the system design is known to change the horizontal resolution (and hence the image size) depending on factors such as scene activity, but only for I-pictures, hence typically for GOP. The resampling of reference pictures to use different resolutions in CVS is known, for example, from ITU-T Rec. H.263, Annex P. However, here the image size is not changed, only the reference pictures are resampled, with the result that only parts of the image canvas can potentially be used (in in the case of downsampling) or capture only parts of the scene (in the case of upsampling). In addition, H.263 Annex Q allows resampling of an individual macroblock by a factor of two (in each dimension), up or down. Again, the image size remains the same. The macroblock size is fixed in H.263 and therefore does not need to be transmitted.

[7] Image resizing in predicted images has become more common in modern video coding. For example, VP9 allows you to resample the reference image and change the resolution for the entire image. Similarly, some of the proposals made in relation to VVC (including, for example, Hendry, et. al., "On adaptive resolution change (ARC) for VVC", JVT document JVET-M0135-v1, January 9-19, 2019, incorporated herein in its entirety) allow resampling of entire reference pictures with a different, higher or lower resolution. This document proposes to encode different resolution candidates in a sequence parameter set and refer to them using the syntax elements for each image in the image parameter set.

DISCLOSURE OF THE INVENTION

[8] Embodiments of the invention relate to a method, system, and computer-readable medium for signaling output layer sets in encoded video data. According to one aspect, a method is provided for signaling output layer sets in encoded video data. The method may include receiving video data having multiple layers. One or more syntax elements are identified. The syntax elements define one or more sets of output layers corresponding to output layers from a plurality of layers of received video data. One or more output layers corresponding to the given sets of output layers are decoded and displayed.

[9] According to another aspect, a computer system is provided for signaling output layer sets in encoded video data. A computer system may include one or more processors, one or more computer-readable storage elements, one or more computer-readable tangible storage devices, and program instructions stored on at least one of one or more storage devices for execution by at least one of one or more more processors through at least one of the one or more memory elements, whereby the computer system is capable of performing the method. The method may include receiving video data having multiple layers. One or more syntax elements are identified. The syntax elements define one or more sets of output layers corresponding to output layers from a plurality of layers of received video data. One or more output layers corresponding to the specified output layer sets are decoded and displayed.

[10] According to another aspect, a computer-readable medium is provided for signaling sets of output layers in encoded video data. The computer-readable medium may include one or more computer-readable memories and program instructions stored on at least one of the one or more tangible storage devices, the program instructions being executed by a processor. The software instructions are executed by the processor to perform the method, which accordingly may include receiving video data having a plurality of layers. One or more syntax elements are identified. The syntax elements define one or more sets of output layers corresponding to output layers from a plurality of layers of received video data. One or more output layers corresponding to the given sets of output layers are decoded and displayed.

BRIEF DESCRIPTION OF THE DRAWINGS

[11] These and other objects, features and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in conjunction with the accompanying drawings. Various elements of the drawings are not shown to scale, as the illustrations are provided for clarity and ease of understanding by a person skilled in the art, together with a detailed description.

On the drawings:

- fig. 1 is a schematic illustration of a simplified block diagram of a communication system in accordance with an embodiment;

- fig. 2 is a schematic illustration of a simplified block diagram of a communication system in accordance with an embodiment;

- fig. 3 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment;

- fig. 4 is a schematic illustration of a simplified block diagram of an encoder according to an embodiment;

- fig. 5 is a schematic illustration of embodiments of ARC parameter signaling in accordance with an embodiment;

- fig. 6 is an example of a syntax table according to an embodiment;

- fig. 7 is a schematic illustration of a computer system in accordance with an embodiment;

- fig. 8 shows an example of a prediction structure for adaptive resolution scalability;

- fig. 9 is an example of a syntax table according to an embodiment;

- fig. 10 is a schematic illustration of a simplified flowchart of parsing and decoding a decrement loop for each access unit and the value of the access unit counter;

- fig. 11 is a schematic illustration of a video bitstream structure containing layered sub-images;

- fig. 12 is a schematic illustration of an enhanced resolution display of a selected sub-image;

- fig. 13 is a flowchart of a process for decoding and displaying a video bitstream containing multilayer sub-images;

- fig. 14 is a schematic illustration of a 360° video display with a sub-image enhancement layer;

- fig. 15 is an example of sub-image layout information and its corresponding layer and image prediction structure;

- fig. 16 shows an example of sub-image layout information and its corresponding layer and image prediction structure with local area spatial scalability modality;

- fig. 17 is an example of a syntax table for sub-image layout information;

- fig. 18 is an example of an SEI message syntax table for sub-image layout information;

- fig. 19 is an example of a syntax table for specifying output layers and profile/tier/level information for each set of output layers;

- fig. 20 is an example of a syntax table for specifying an output layer mode on each set of output layers; And

- fig. 21 is an example of a syntax table for specifying the current subimage of each layer for each set of output layers.

IMPLEMENTATION OF THE INVENTION

[12] Detailed embodiments of the claimed structures and methods are disclosed in this description; however, it can be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which may be embodied in various forms. However, these structures and methods can be embodied in many different forms and should not be construed as being limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this description is complete and complete and fully conveys the scope of legal protection for those skilled in the art. In the description, details of well-known functions and methods may be omitted in order to avoid unnecessarily obscuring the understanding of the presented embodiments.

[13] Embodiments generally relate to the field of data processing and, in particular, to the processing of multimedia. The exemplary embodiments described below disclose a system, method, and computer program to, among other things, signal coded video data output layer sets. Therefore, some embodiments may improve the computational domain through improved video encoding and decoding.

[14] As previously described, video encoders and video decoders tended to work with a given picture size, which in most cases was defined and held constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2, the system design is known to change the horizontal resolution (and hence the image size) depending on factors such as scene activity, but only for I-pictures, hence typically for GOP. The resampling of reference pictures to use different resolutions in CVS is known, for example, from ITU-T Rec. H.263, Annex P. However, here the image size is not changed, only the reference pictures are resampled, with the result that only parts of the image canvas can potentially be used (in in the case of downsampling) or capture only parts of the scene (in the case of upsampling). In addition, H.263 Annex Q allows resampling of an individual macroblock by a factor of two (in each dimension), up or down. Again, the image size remains the same. The macroblock size is fixed in H.263 and therefore does not need to be signaled.

[15] However, in the context of e.g. 360 encoding or certain surveillance applications, multiple semantically independent source images (e.g., the surface of six cubes of a cube-projected scene 360, or individual camera inputs in the case of a multi-camera surveillance setup) may require separate adaptive resolution settings, to handle the different activity of each scene at a given time. In other words, encoders at a given point in time may choose to use different resampling factors for different semantically independent pictures that make up the entire 360-degree or observation scene. When merged into a single picture, in turn, there is a need to perform resampling of the reference picture and to have adaptive resolution coding signaling available for portions of the encoded picture. Therefore, it may be preferable to use the available adaptive resolution coding signaling data for better signaling, coding, decoding, and display of video layers.

[16] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. System (100) may include at least two terminals (110-120) interconnected via network (150). For unidirectional data transmission, the first terminal (110) may encode video data at the local location for transmission to another terminal (120) via the network (150). The second terminal (120) may receive the encoded video data of another terminal from the network (150), decode the encoded data, and display the recovered video data. Unidirectional data transfer may be common in media serving applications and the like.

[17] FIG. 1 shows a second pair of terminals (130, 140) providing support for bidirectional encoded video transmission, such as may occur during a video conference. For bi-directional data transmission, each terminal (130, 140) may encode video data captured at a local location for transmission to another terminal over a network (150). Each terminal (130, 140) may also receive the encoded video data transmitted by the other terminal, may decode the encoded data, and may display the recovered video data on a local display device.

[18] FIG. 1, terminals (110-140) may be depicted as servers, personal computers, and smartphones, but the principles of the present disclosure cannot be limited to this. Embodiments of the present invention find application in laptop computers, tablet computers, media players and/or specialized video conferencing equipment. Network (150) is any number of networks that transmit encoded video data between terminals (110-140), including, for example, wired and/or wireless networks. The communication network (150) may communicate on circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks and/or the Internet. For the purposes of this discussion, the architecture and topology of the network (150) may not be essential to the operation of this disclosure, unless explained herein below.

[19] FIG. 2 illustrates, as an example of application for the disclosed subject matter, the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, videoconferencing, digital television, storage of compressed video on digital media including CD, DVD, memory card, etc., and so on. .

[20] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a digital camera, producing, for example, a stream (202) of uncompressed video samples. This stream (202) of samples, depicted with a thick line to emphasize the large amount of data compared to encoded video bitstreams, can be processed by an encoder (203) connected to a camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (204), drawn with a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server (205) for future use. One or more streaming clients (206, 208) can access the streaming server (205) to extract copies (207, 209) of the encoded video bitstream (204). The client (206) may include a video decoder (210) that decodes an incoming copy of the encoded video bitstream (207) and produces an outgoing stream (211) of video samples that can be displayed on a display (212) or other visualization device (not shown). In some streaming systems, the video bitstreams (204, 207, 209) may be encoded according to certain video coding/compression standards. Examples of these standards include ITU-T Rec. H.265. A video coding standard is under development, informally known as Universal Video Coding or VVC. The expanded object can be used in a VVC context.

[21] FIG. 3 may be a functional block diagram of a video decoder (210) according to one or more embodiments.

[22] The receiver (310) may receive one or more encoded video sequences to be decoded by the decoder (210); in the same or another embodiment, one coded video sequence at a time, where the decoding of each coded video sequence is independent of other coded video sequences. An encoded video sequence may be received from a channel (312), which may be a hardware/software connection to a storage device that stores the encoded video data. The receiver (310) may receive encoded video data with other data, such as encoded audio data and/or ancillary data streams, which may be sent to their respective using entities (not shown). The receiver (310) may separate the encoded video sequence from other data. To combat network jitter, a buffer memory (315) can be connected between the receiver (310) and the entropy decoder/parser (320) (hereinafter "parser"). When the receiver (310) receives data from a storage/transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, buffer memory (315) may not be needed or may be small. For use in best quality packet networks such as the Internet, a buffer (315) may be required, which may be relatively large and may preferably be of an adaptive size.

[23] The video decoder (210) may include a parser (320) for recovering the symbols (321) from the entropy coded video sequence. The categories of these symbols include information used to control the operation of the decoder (210) and potentially information to control a visualization device such as a display (212) that is not an integral part of the decoder but can be connected to it as shown. in fig. 2. The control information for the rendering device(s) may be in the form of additional enhancement information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser (320) may parse/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be in accordance with a video coding technology or standard, and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The parser (320) may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), and so on. The entropy decoder/parser may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[24] The parser (320) may perform an entropy decoding/parsing operation of the video sequence received from the buffer (315) to create symbols (321).

[25] Symbol recovery (321) may include many different modules depending on the type of encoded video image or parts thereof (such as: external and internal image, external and internal block) and other factors. Which modules are involved and how can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (320). The flow of such subgroup control information between the parser (320) and the plurality of modules is not shown below for clarity.

[26] In addition to the functional blocks already mentioned, the decoder 210 can be conceptually subdivided into a number of functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with each other and can be at least partially integrated into each other. However, for purposes of describing the disclosed subject matter, the conceptual division into functional modules below is appropriate.

[27] The first module is the scaling/inverse transform module (351). The scaling/inverse transform module (351) receives the quantized transform coefficient, as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, and so on, in the form of symbol(s) (321) from the parser ( 320). It can output blocks containing sample values that can be entered into the aggregator (355).

[28] In some cases, the output samples of the scaling/inverse transform module (351) may refer to an intra-coded block; that is, to a block that does not use prediction information from previously restored images, but can use prediction information from previously restored portions of the current image. Such prediction information may be provided by the intra prediction module (352). In some cases, the intra-prediction module (352) generates a block of the same size and shape as the reconstructed block using the already reconstructed surrounding information extracted from the current (partially reconstructed) image (356). The aggregator (355), in some cases, adds for each sample the prediction information generated by the intra-prediction module (352) to the output sample information provided by the scaling/inverse transform module (351).

[29] In other cases, the output samples of the scaling/inverse transform module (351) may refer to a block with intercoding and potentially with motion compensation. In such a case, the motion compensation prediction module (353) may access the reference picture memory (357) to retrieve samples used for prediction. After compensating for the movement of the selected samples according to the symbols (321) associated with the block, these samples can be added by the aggregator (355) to the output of the scaler/demap module (in this case called residual samples or residual signal) to generate output information. readings. Reference picture memory addresses from where the motion compensation prediction module selects prediction samples may be driven by motion vectors available to the motion compensation prediction module in the form of symbols (321), which may have X, Y and reference picture components, for example. Motion compensation may also include interpolation of sample values retrieved from reference picture memory when accurate sub-sample motion vectors are used, motion vector prediction mechanisms, and so on.

[30] The output samples of the aggregator (355) may be subjected to various loop filtering methods in the loop filter module (356). Video compression technologies may include in-loop filtering technologies that are controlled by parameters included in the encoded video bitstream and provided to the loop filtering module (356) as symbols (321) from the parser (320), but may also respond to metainformation received during decoding the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as responding to previously restored and filtered by loop filtering sample values.

[31] The output of the loop filter module (356) may be a stream of samples that can be output to the renderer (212) and also stored in the reference picture memory (356) for use in future inter-picture prediction.

[32] Certain encoded pictures after full recovery can be used as reference pictures for future prediction. Once the encoded picture has been completely reconstructed and the encoded picture has been identified as a reference picture (eg, by the parser (320)), the current reference picture (356) may become part of the reference picture buffer (357), and fresh current picture memory may be reallocated before starting restoring the next encoded picture.

[33] Video decoder 320 may perform decoding operations according to a predetermined video compression technology that may be written in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax given by the video compression technology or standard used, in the sense that it conforms to the syntax of the video compression technology or standard as specified in the video compression technology document or standard, and in particular their profile document. It may also be necessary for compliance that the complexity of the encoded video sequence be within limits defined by the level of technology or video compression standard. In some cases, levels limit the maximum image size, the maximum frame rate, the maximum recovery sampling rate (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the layers may, in some cases, be further limited by hypothetical reference decoder (HRD) specifications and metadata for managing the HRD buffer transmitted in the encoded video sequence.

[34] In an embodiment, the receiver (310) may receive additional (redundant) data with encoded video. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (320) to correctly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[35] FIG. 4 may be a functional block diagram of a video encoder (203) according to an embodiment of the present disclosure.

[36] The encoder (203) may receive video samples from a video source (201) (which is not part of the encoder) that may capture video image(s) for encoding by the encoder (203).

[37] The video source (201) may provide the source video sequence for encoding by the encoder (203) in the form of a digital stream of video samples, which may have any suitable bit depth (eg: 8 bit, 10 bit, 12 bit, ...), any color space (eg BT.601 Y CrCB, RGB, ...) and any suitable sample structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (201) may be a storage device storing pre-prepared video. In a videoconferencing system, the video source (203) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual images that, when viewed sequentially, convey motion. The images themselves can be organized as a spatial array of pixels, in which each pixel can contain one or more samples depending on the sample structure used, color space, and so on. One skilled in the art can easily understand the relationship between pixels and samples. The description below is based on readings.

[38] According to an embodiment, the encoder (203) may encode and compress images of the original video sequence into the encoded video sequence (443) in real time or with any other time constraints as required by the application. Providing an appropriate coding rate is one of the functions of the controller (450). The controller controls other functional modules, as described below, and is functionally associated with these modules. The connection is not shown for clarity. The parameters set by the controller may include parameters related to speed control (picture skip, quantizer, speed-distortion optimization methods lambda value, ...), image size, group of pictures (GOP) layout, maximum motion vector search range, and so on. . A person skilled in the art can easily determine other functions of the controller (450) as they may relate to a video encoder (203) optimized for a particular system design.

[39] Some video encoders operate in what one of ordinary skill in the art would readily recognize as a "coding loop". As a simplified description, an encoding loop may consist of an encoder coding part (430) (hereinafter "source encoder") (responsible for creating symbols based on the input picture to be encoded and the reference picture(s)), and a (local) decoder (433) built into the encoder (203) that decompresses the symbols to create sample data that the (remote) decoder can also create (because any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies considered in the disclosed subject matter) . This reconstructed sample stream is entered into the reference picture memory (434). Because decoding a stream of symbols results in bit-accurate results, regardless of the location of the decoder (local or remote), the contents of the reference picture buffer are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" using prediction during decoding. This fundamental principle of reference picture synchronism (and the resulting drift if synchronism cannot be maintained due to, for example, channel errors) is well known to those skilled in the art.

[40] The operation of the "local" decoder (433) may be the same as that of the "remote" decoder (210), which has already been described in detail above with reference to FIG. 3. Briefly referring also to FIG. 3, however, since the symbols are available and encoding/decoding of the symbols into the encoded video sequence by the entropy encoder (445) and the parser (320) can be lossless, the entropy decoding parts of the decoder (210), including the channel (312), the receiver (310), buffer (315) and parser (320) cannot be fully implemented in the local decoder (433).

[41] At this point, it can be observed that any decoding technology other than parsing/entropy decoding that is present in a decoder must also necessarily be present in a substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on the operation of the decoder. The description of encoding technologies can be abbreviated as they are the inverse of fully described decoding technologies. Only in some areas a more detailed description is required, which is given below.

[42] As part of its operation, the source encoder (430) may perform motion-compensated predictive encoding that predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence, which have been designated "reference frames". Thus, the coding engine (432) encodes the differences between the pixel blocks of the input frame and the pixel blocks of the reference frame(s) that can be selected as the prediction reference(s) for the input frame.

[43] The local video decoder (433) may decode the encoded video data of the frames, which may be designated as reference frames, based on the symbols generated by the source encoder (430). The operations of the encoding engine (432) may preferably be lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (433) replicates the decoding processes that may be performed by the video decoder on the reference frames and may cause the recovered reference frames to be stored in the reference picture cache (434). Thus, the encoder (203) can locally store copies of the reconstructed reference frames that have common content as reconstructed reference frames to be received by the far-end video decoder (in the absence of transmission errors).

[44] The predictor (435) may perform predictive searches for the encoding engine (432). That is, for a new picture to be encoded, the predictor (435) may search the reference picture memory (434) for sample data (as reference pixel block candidates) or certain metadata such as reference picture motion vectors, block shapes, and so on. further, which can serve as a suitable prediction reference for new images. The predictor (435) may operate on a block-by-pixel basis to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (434).

[45] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[46] The output signals of all the above functional modules can be entropy encoded in an entropy encoder (445). The entropy encoder translates symbols generated by various functional modules into an encoded video sequence by lossless symbol compression according to techniques known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, and so on.

[47] The transmitter (440) may buffer the encoded video sequence(s) generated by the entropy encoder (445) to prepare it for transmission over a communication channel (460), which may be a hardware/software connection to a storage device that will store the encoded video data. . The transmitter (440) may merge the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or ancillary data streams (sources not shown).

[48] The controller (450) may control the operation of the encoder (203). During encoding, the controller (450) may assign to each encoded picture a certain type of encoded picture, which may affect the encoding methods that can be applied to the corresponding picture. For example, images can often be assigned to one of the following frame types:

[49] An intra-picture (I-picture) may be one that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow various types of internal images, including, for example, images with independent decoder update. The person skilled in the art will be aware of these variants of I-images and their respective uses and features.

[50] A predictive picture (P-picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most one motion vector and a reference index to predict sample values of each block.

[51] A bidirectional predictive picture (B-picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most two motion vectors and reference indices to predict sample values of each block. Similarly, multi-prediction images may use more than two reference images and associated metadata to reconstruct a single block.

[52] Source images can typically be spatially divided into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. The blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the coding assignment applied to the corresponding block images. For example, I-picture blocks may be predictively encoded or they may be predictively encoded with reference to already encoded blocks of the same picture (spatial prediction or intra prediction). P-picture pixel blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one previously encoded reference picture. Blocks of B-pictures may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one or two previously encoded reference pictures.

[53] The video encoder (203) may perform encoding operations in accordance with a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (203) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore follow the syntax given by the video coding technology or standard used.

[54] In an embodiment, transmitter (440) may transmit additional data with encoded video. Video encoder (430) may include such data as part of an encoded video sequence. The additional data may comprise temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant images and slices, enhancement supplemental information (SEI) messages, video usability information (VUI) parameter set fragments, and so on.

[55] Before a more detailed description of certain aspects of the disclosed subject matter, it is necessary to introduce a few terms that will be referred to in the remainder of this description.

[56] A sub-image further refers to, in some cases, a rectangular arrangement of samples, blocks, macroblocks, coding units, or the like, which are semantically grouped and which can be independently re-resolution encoded. One or more subimages may be used for an image. One or more encoded sub-images may form an encoded image. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In certain environments, one or more encoded sub-images can be collected in the compressed area without re-coding to the sample level in the encoded image, and in the same or some other cases, one or more encoded sub-images can be extracted from the encoded image in the compressed area.

[57] Hereinafter, adaptive resolution change (ARC) refers to mechanisms that allow changing the resolution of a picture or sub-picture in a coded video sequence, for example, by resampling a reference picture. Hereinafter, ARC parameters refer to control information necessary to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, output and/or reference picture resolutions, various control flags, and so on.

[58] The above description focuses on the encoding and decoding of a single semantically independent encoded video image. Before describing the consequences of encoding/decoding multiple sub-images with independent ARC parameters and their implied additional complexity, signaling options for the ARC parameters must be described.

[59] FIG. 5 shows several new options for signaling ARC parameters. As noted for each of the options, they have certain advantages and certain disadvantages in terms of coding efficiency, complexity and architecture. A video coding standard or technology may choose one or more of these or prior art options for transmitting the ARC parameters. The options may not be mutually exclusive and may possibly be interchangeable depending on the needs of the application, the standards used, or the choice of encoder.

[60] ARC parameter classes may include:

upsampling/downsampling factors, either individually or combined across the X and Y dimensions.

upsampling/downsampling factors with the addition of a temporal dimension, indicating an increase/decrease in size at a constant rate for a given number of images.

Either of the above two options may include encoding one or more supposedly short syntax elements that may point to a table containing the coefficient(s).

resolution in X or Y dimension in units of samples, blocks, macroblocks, CUs, or any other suitable granularity of the input image, output image, reference image, coded image, combined or individually. If there is more than one resolution (eg, one for the input image, one for the reference image), then in some cases one set of values may be derived from another set of values. This can be communicated, for example, by using flags. See below for a more detailed example.

"Warp" coordinates like those used in H.263 Annex P, again with the appropriate granularity as described above. Annex P H.263 defines one efficient way to encode such warp coordinates, but other potentially more efficient ways may also be developed. For example, the reverse Huffman encoding with variable length warp coordinates according to Annex P can be replaced by binary encoding of an appropriate length, where the length of the binary codeword can be, for example, obtained from the maximum image size, perhaps multiplied by a certain factor and shifted by a certain value, so that take into account "warping" beyond the boundaries of the maximum image size, upsampling or downsampling filter options. In the simplest case, there can be only one filter for upsampling and/or downsampling. However, in some cases it may be desirable to provide more flexibility in filter design, and this may require filter parameter signaling. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g., by a list of filter coefficients using suitable entropy coding techniques), the filter may be implicitly selected by appropriate upsampling/downsampling relationships that , in turn, are transmitted according to any of the mechanisms mentioned above, and so on.

[61] The following description assumes encoding a finite set of upsampling/downsampling coefficients (the same coefficient to be used in the X and Y dimensions) indicated by a codeword. This codeword may preferably be encoded with a variable length, for example using the Exponential Golomb code common to certain syntax elements in video coding specifications such as H.264 and H.265.

[62] Many similar mappings can be developed according to the needs of the application and the capabilities of the upsampling and downsampling mechanisms available in the video compression technology or standard. The table can be expanded to more values. Values may also be represented by entropy encoding mechanisms other than Exponential Golomb codes, such as using binary encoding. This may have certain advantages when the oversampling factors were of interest outside of the video processing engines themselves (primarily the encoder and decoder), for example via MANE. It should be noted that for the (presumably) most common case where no resolution change is required, Exponential Golomb code, which is short, can be chosen; there is only one bit in the table above. This may have an advantage in coding efficiency over using binary codes for the most common case.

[63] The number of entries in the table, as well as their semantics, can be completely or partially customizable. For example, the basic table structure may be conveyed in a "high" parameter set, such as a sequence or a decoder parameter set. Alternatively, or in addition, one or more such tables may be defined in a video coding technology or standard and may be selected, for example, by a decoder or sequence parameter set.

[64] The following describes how the upsampling/downsampling ratio (ARC information) encoded as described above can be included in a video coding technology or a standard syntax. Similar considerations may apply to one or more codewords driving the upsampling/downsampling filters. See the discussion below for when a filter or other data structure requires relatively large amounts of data.

[65] Appendix P H.263 includes ARC information 502 in the form of four warp coordinates in the image header 501, specifically in the PLUSPTYPE (503) H.263 header extension. This may be a reasonable design choice when a) there is an image header available and b) frequent changes to the ARC information are expected. However, the overhead when using H.263 style transmission can be quite high, and the scaling factors may not be related to the image borders, since the image header may be temporary.

[66] The JVCET-M135-v1 cited above includes the ARC reference information (505) (index) located in the image parameter set (504), indexes the table (506) including target resolutions, which in turn is located within a set (507) of sequence parameters. Placing a candidate resolution in a table (506) in a sequence parameter set (507) may, in accordance with the verbal statements made by the authors, be justified by using the SPS as a compatibility negotiation point during capability exchange. The resolution can be changed within the limits set by the values in the table (506) from image to image by referring to the corresponding set (504) of image parameters.

[67] Still referring to FIG. 5, the following additional options may exist for transmitting ARC information in a video bitstream. Each of these options has certain advantages over the current state of the art, as described above. Options may be present in the same video coding technology or standard at the same time.

[68] In an embodiment, ARC information (509) such as a resampling (scaling) factor may be present in a slice header, GOB header, tile header, or tile group header (hereinafter tile group header) (508). This may be sufficient for small size ARC information such as a single variable length codeword ue(v) or a fixed length codeword of multiple bits, such as shown above. Having the ARC information directly in the tile group header has the additional benefit that the ARC information may be applicable to a sub-image represented by that tile group, for example, rather than to the entire image. See also below. Furthermore, even if a video compression technology or standard only provides for changing the adaptive resolution of the entire image (as opposed to adaptive resolution changes based on a group of tiles, for example), placing the ARC information in the tile group header instead of placing it in the H-style image header. 263 has certain advantages in terms of error tolerance.

[69] In the same or another embodiment, the ARC information (512) itself may be present in the corresponding parameter set (511), such as, for example, a picture parameter set, a header parameter set, a tile parameter set, an adaptation parameter set, and so on ( a set of adaptation parameters is shown). The scope of this set of parameters may preferably be no larger than an image, such as a group of tiles. The use of ARC information is implicitly done by activating the appropriate set of options. For example, when a video coding technology or standard considers only image-based ARC, then an image parameter set or equivalent may be appropriate.

[70] In the same or another embodiment, the ARC reference information (513) may be present in a tile group header (514) or a similar data structure. This reference information (513) may refer to a subset of ARC information (515) available in a parameter set (516) with a scope larger than one picture, such as a sequence parameter set or a decoder parameter set.

[71] The additional level of indirection implying the activation of PPS from the tile group header, PPS, SPS, which is used in JVET-M0135-v1, seems unnecessary, since image parameter sets, like sequence parameter sets, can be used (and are used in some standards such as RFC3984) to negotiate capabilities or declarations. However, if the ARC information is to be applicable to a sub-image represented, for example, also by tile groups, a parameter set with an activation area limited by the tile group, such as an adaptation parameter set or a header parameter set, may be a better choice. Also, if the ARC information is of more than a small size, such as containing filter control information such as multiple filter coefficients, then the parameter may be a better choice than using the (508) header directly from a coding gain point of view, as these settings may be reused in future images or subimages, referring to the same set of parameters.

[72] When using a sequence parameter set or other higher-level parameter set with a scope spanning multiple images, certain considerations may apply:

[73] The parameter set for storing the ARC information table (516) may in some cases be a sequence parameter set, but in other cases preferably a decoder parameter set. A decoder parameter set may have an activation area of several CVSs, namely an encoded video stream, i. e. all encoded bits of the video from the beginning to the end of the session. Such scope may be more appropriate because the possible ARC coefficients may be a function of the decoder, possibly implemented in hardware, and hardware functions generally do not change with any CVS (which, at least in some entertainment systems, is a group images one second or less in length). However, the inclusion of a table in a sequence parameter set is explicitly included in the placement options described here.

[74] The ARC reference information (513) can preferably be placed directly in the image/tile slice/GOB/tile group header (514) (hereinafter the tile group header) rather than in the image parameter set as in JVCET-M0135-v1. The reason is that when an encoder wants to change one value in a picture parameter set, such as the ARC reference information, for example, it must create a new PPS and refer to this new PPS. Assume that only the ARC reference information is changed, but other information such as quantization matrix information in PPS, for example, remains. Such information may be of considerable size and will need to be retransmitted in order to complete the new PPS. Since the ARC reference information may be a single codeword, such as an index in a table (513), and this will be the only value that changes, it would be cumbersome and wasteful to retransmit, for example, all of the quantization matrix information. In this regard, it may be significantly better in terms of coding efficiency to avoid the PPS indirection as proposed in JVET-M0135-v1. Similarly, placing the ARC reference information in the PPS has the additional disadvantage that the ARC information referred to by the ARC reference information (513) must necessarily be applied to the whole picture and not to the sub-picture, since the activation scope of the picture parameter set represents is an image.

[75] In the same or another embodiment, ARC parameter signaling may follow the detailed example as shown in FIG. 6. In FIG. 6 shows syntax diagrams in the representation used in video coding standards since at least 1993. The notation for such syntax diagrams roughly corresponds to C-style programming. Lines in bold indicate syntax elements present in the bitstream, lines without bold often indicate control flow or setting variables.

[76] A tile group header (601) as an exemplary header syntax structure applicable to a (possibly rectangular) part of an image, may conditionally contain a variable length syntax element dec_pic_size_idx (602) encoded with the exponential Golomb code (bolded). The presence of this syntax element in a tilegroup header can be restricted using adaptive resolution (603) - here the flag value is not bolded, meaning that the flag is present in the bitstream at the point where it occurs in the syntax diagram. Whether adaptive resolution is used for this image or parts of it can be signaled in any high-level syntactic structure inside or outside the bitstream. In the example shown, this is signaled in the sequence parameter set as shown below.

[77] Still referring to FIG. 6, a selection of a set (610) of sequence parameters is also shown. The first syntax element shown is adaptive_pic_resolution_change_flag (611). When set to true, this flag may indicate the use of adaptive resolution, which in turn may require certain control information. In the example, such control information is conditionally present based on the value of the flag based on the if() statement in the parameter set (612) and the tile group header (601).

[78] When adaptive resolution is used, the output resolution is encoded in units of samples (613) in this example. Position number 613 refers to both output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can determine the resolution of the output image. Elsewhere in a video coding technology or standard, certain restrictions may be defined for any value. For example, the level definition may limit the number of total output samples that can be the product of the values of these two syntax elements. Also, certain video coding technologies or standards, or external technologies or standards such as, for example, system standards, may restrict the numbering range (for example, one or both dimensions must be divisible by a power of 2) or the ratio of width and height (for example, width and height should be in a ratio such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementations or for other reasons, and are well known in the art.

[79] In some applications, it may be desirable for the encoder to instruct the decoder to use a specific reference picture size instead of implicitly assuming that this size is the size of the output picture. In this example, the syntax element reference_pic_size_present_flag (614) conveys the conditional presence of the dimensions (615) of the reference picture (again, the position number refers to both width and height).

[80] Finally, a table of possible decoding image width and height is shown. Such a table may be expressed, for example, by specifying a table (num_dec_pic_size_in_luma_samples_minus1) (616). "Minus1" may refer to the interpretation of the meaning of this syntax element. For example, if the encoded value is zero, there is one entry in the table. If the value is five, there are six table entries. Then, for each "row" in the table, the width and height of the decoded picture are included in the syntax (617).

[81] Represented table entries (617) can be indexed using the dec_pic_size_idx (602) syntax element in the tile group header, thereby allowing different decoded sizes—essentially scaling factors—for each tile group.

[82] The techniques for transmitting adaptive resolution parameters described above may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, in FIG. 7 shows a computer system 700 suitable for implementing certain embodiments of the disclosed subject matter.

[83] Computer software may be coded using any suitable machine code or computer language, which may be the object of assembly, compilation, linking, or similar mechanisms to produce code containing instructions that can be executed directly or through interpretation, execution of microcode, etc. .p., computer central processing units (CPUs), graphics processing units (GPUs), etc.

[84] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[85] The components shown in FIG. 7 for computer system 700 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of computer software implementing embodiments of the present disclosure. The component configuration is also not to be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the exemplary embodiment of computer system 700.

[86] Computer system 700 may include certain human-machine interface input devices. Such a human-machine interface input device may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a control glove), audio input (e.g., voice, handclaps), visual input (for example: gestures), olfactory input (not shown). HMI devices can also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, ambient sound), images (e.g. scanned images, photographic images taken from a camera for still images), video (eg 2D video, 3D video including stereoscopic video).

[87] Human-machine interface input devices may include one or more of (only one of each depicted): keyboard 701, mouse 702, trackpad 703, touch screen 710, control glove 704, joystick 705, microphone 706, scanner 707, camera 708.

[88] Computer system 700 may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human interface output devices may include tactile output devices (e.g., haptic feedback from a 710 touch screen, 704 control glove, or 705 joystick, but there may also be tactile feedback devices that do not serve as input devices) , audio output devices (for example: speakers 709, headphones (not shown)), image output devices (for example, screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each of which has or does not have the ability to touch screen input, each with or without tactile feedback capability - some of which may be capable of producing 2D visual output or more than 3D output by means such as stereographic output; virtual reality glasses (not shown), holographic displays and smoke tanks (not shown) and printers (not shown).

[89] Computer system 700 may also include human accessible storage devices and associated media such as optical media including CD/DVD ROM/RW 720 with CD/DVD or similar media 721, flash drive 722, removable hard disk or solid state drive 723, previously developed magnetic media such as tape and floppy disk (not shown), dedicated ROM/ASIC/PLD based devices such as dongles (not shown), etc.

[90] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other time signals.

[91] Computer system 700 may also include an interface to one or more communication networks. Networks can be, for example, wireless, wired, optical. In addition, networks can be local, wide area, metropolitan, automotive, industrial, real-time, latency tolerant, and so on. Example networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired television or wireless digital wide area networks including cable television, satellite television and terrestrial TV broadcasting, automotive and industrial equipment, including CAN bus, etc. Some networks typically require external network interface adapters that are connected to certain general purpose data ports or peripheral buses (749) (for example, the USB ports of a 700 computer system); others are typically integrated into the core of computer system 700 by being attached to a system bus as described below (eg, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 700 can communicate with other entities. Such communication can be unidirectional, receive-only (for example, broadcast television), unidirectional, send-only (for example, CAN bus to certain CAN-bus devices), or bidirectional, for example, to other computer systems using a local or wide area digital network. . Each of these networks and network interfaces may use specific protocols and protocol stacks as described above.

[92] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core 740 of the computer system 700.

[93] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, dedicated field programmable gate array (FPGA) processors 743, hardware accelerators for certain tasks 744, etc. . These devices, along with Read Only Memory (ROM) 745, Random Access Memory 746, mass storage internal storage such as internal non-user hard drives, solid state drives (SSDs), and the like. 747 may be connected via system bus 748. In some computer systems, system bus 748 may be available as one or more physical connectors to allow expansion with additional CPUs, GPUs, and the like. Peripherals may be connected either directly to the kernel system bus 748 or via a peripheral bus 749. Peripheral bus architectures include PCI, USB, and the like.

[94] CPU 741, GPU 742, FPGA 743, and accelerators 744 may execute certain instructions, which in combination may constitute the aforementioned computer code. This computer code may be stored in ROM 745 or RAM 746. Transient data may also be stored in RAM 746, while persistent data may be stored in, for example, internal mass storage 747. Fast storage and retrieval to any of the storage devices may be accessed by the use of cache memory, which may be closely associated with one or more of the CPU 741, GPU 742, mass storage 747, ROM 745, RAM 746, and the like.

[95] The computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specially designed and constructed for the purposes of this disclosure, or they may be well known and available to those skilled in the art of computer software.

[96] By way of example, and not limitation, a computer system having architecture 700, and in particular core 740, may provide functionality as a result of having processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied on one or more tangible, computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as certain core storage devices 740 that are of a non-transitory nature, such as internal mass storage 747 or ROM 745. implementation of the present disclosure may be stored in such devices and executed by the core 740. The computer-readable medium may include one or more memories or chips in accordance with specific needs. The software may cause the core 740, and in particular the processors therein (including the CPU, GPU, FPGA, etc.) to execute certain processes or certain parts of the specific processes described herein, including the definition of data structures stored in RAM 746 and modifying such data structures in accordance with processes defined by the software. In addition or alternatively, a computer system may provide functionality as a result of logic embedded in hardware circuitry or otherwise embodied in circuitry (e.g., an accelerator 744) that can operate in place of or in conjunction with software to perform certain processes or parts of specific processes described in this document. The link to the software may include logic and vice versa where applicable. A reference to a computer-readable medium may include circuitry (eg, an integrated circuit (IC)) storing software to execute, circuitry embodying logic to execute, or both, where applicable. The present disclosure includes any suitable combination of hardware and software.

[97] Some video coding technologies or standards, such as VP9, support spatial scalability by implementing certain forms of reference picture resampling (transmitted quite differently than in the disclosed subject matter) in combination with temporal scalability to provide spatial scalability. In particular, some reference pictures may be upsampled using ARC-style techniques to higher resolution to form the base of the spatial enhancement layer. These upsampled images can be enhanced using normal high resolution prediction engines to add detail.

[98] The disclosed subject matter may be used in such an environment. In some cases, in the same or another embodiment, a value in the NAL unit header, such as a temporal identifier field, may be used to indicate not only the temporal but also the spatial layer. This offers certain advantages for certain system configurations; for example, an existing Selected Forwarding Units (SFU) architecture created and optimized for selected temporal slices forwarded based on the NAL Unit Header Temporal Identifier value can be used without modification for scalable environments. In order to make this possible, there may be a requirement for a mapping between the encoded picture size and the temporal layer indicated by the temporal identifier field in the NAL unit header.

[99] In some video coding technologies, an access unit (AU) may refer to the encoded image(s), slice(s), tile(s), NAL unit(s), etc., that have been captured and arranged into the corresponding image/ slice/tile/bitstream of the NAL unit at a given time. This point in time may be link time.

[100] In HEVC and some other video coding technologies, a picture order counter (POC) value may be used to indicate a selected reference picture among a plurality of reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) contains one or more images, slices or tiles, each image, slice or tile belonging to the same AU can carry the same POC value, from which it can be concluded that they were created from content at the same link time. In other words, in a scenario where two images/slices/tiles carry the same POC setpoint, this may indicate that the two images/slices/tiles belong to the same AU and have the same link time. Conversely, two images/slices/tiles having different POC values may indicate those images/slices/tiles that belong to different AUs and have different link times.

[101] In an embodiment of the disclosed subject matter, the aforementioned rigid relationship can be relaxed because an access unit may contain images, slices, or tiles with different POC values. By allowing different POC values in the AU, it becomes possible to use the POC value to identify potentially independently decodable pictures/slices/tiles with the same presentation time. This, in turn, can support multiple scalable layers without changing the reference picture selection signaling (eg, reference picture set signaling or reference picture list signaling), as described in more detail below.

[102] However, it is still desirable to be able to identify the AU that owns an image/slice/tile with respect to another image/slice/tile having different POC values based on the POC value alone. This can be achieved as described below.

[103] In the same or other embodiments, an access unit counter (AUC) may be signaled in a high-level syntax structure such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or AU delimiter. The AUC value may be used to identify which NAL units, pictures, slices or tiles belong to a given AU. The AUC value may correspond to a particular link time. The AUC value may be a multiple of the ROS value. By dividing the POC value by an integer value, the AUC value can be calculated. In some cases, division operations may impose a certain burden on decoder implementations. In such cases, slight restrictions on the numbering space of the AUC values may allow the division operation to be replaced by shift operations. For example, the AUC value may be equal to the most significant bit (MSB) value of the POC value range.

[104] In the same embodiment, the per-AU ROS cycle value (poc_cycle_au) may be signaled in a high-level syntax structure such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or AU delimiter. The ROS cycle au value may indicate how many different and consecutive ROS values can be associated with the same AU. For example, if the poc_cycle_au value is 4, images, slices, or tiles with a POC value of 0-3 inclusive are associated with an AU with an AUC value of 0, and images, slices, or tiles with a POC value of 4-7, inclusive, are associated with an AU with an AUC value of 1. Therefore, the AUC value can be derived by dividing the POC value by the poc_cycle_au value.

[105] In the same or another embodiment, the poc_cyle_au value may be derived from information located, for example, in a video parameter set (VPS) that identifies the number of spatial or SNR layers in the encoded video sequence. Such a possible relationship is briefly described below. Although derivation as described above may save a few bits in the VPS and therefore may increase coding efficiency, it may be preferable to explicitly encode poc_cycle_au in an appropriate high-level syntax hierarchically below the video parameter set in order to be able to minimize poc_cycle_au for a given small portion of the bitmap. a stream such as an image. This optimization can save more bits than can be stored by the derivation process described above because POC values (and/or POC indirectly related syntax element values) can be encoded in low level syntax structures.

[106] FIG. 8 shows an example of a video sequence structure with a combination of temporal_id, layer_id, POC, and AUC values with adaptive resolution change. In this example, an image, slice or tile in the first AU with AUC=0 may have temporal_id=0 and layer_id=0 or 1, while an image, slice or tile in the second AU with AUC=1 may have temporal_id=1 and layer_id =0 or 1 respectively. The POC value is incremented by 1 for each image regardless of the temporal_id and layer_id values. In this example, the value of poc_cycle_au may be equal to 2. Preferably, the value of poc_cycle_au may be set equal to the number of layers (spatial scalability). Therefore, in this example, the ROS value is increased by 2 and the AUC value is increased by 1.

[107] In the above embodiments, all or a subset of the inter-picture or inter-layer prediction structure and reference picture indication may be supported using existing reference picture set (RPS) signaling in HEVC or reference picture list (RPL) signaling. In RPS or RPL, the selected reference picture is indicated by signaling a POC value or a delta POC value between the current picture and the selected reference picture. For the disclosed subject matter, RPS and RPL may be used to indicate an inter-picture or inter-layer prediction structure without changing the signaling, but with the following limitations. If the temporal_id value of the reference picture is greater than the temporal_id value of the current picture, the current picture may not use the reference picture for motion compensation or other predictions. If the layer_id value of the reference picture is greater than the layer_id value of the current picture, the current picture may not use the reference picture for motion compensation or other predictions.

[108] In the same and other embodiments, motion vector scaling based on the POC difference for temporal motion vector prediction may be disabled for multiple pictures in an access unit. Therefore, although each picture may have a different POC value in the access unit, the motion vector is not scaled and is used for temporal prediction of the motion vector in the access unit. This is because a reference picture with a different ROS in the same AU is considered a reference picture having the same point in time. Therefore, in an embodiment, the motion vector scaling function may return 1 when the reference picture belongs to the AU associated with the current picture.

[109] In the same and other embodiments, motion vector scaling based on the POC difference for temporal motion vector prediction can be optionally disabled for multiple images when the spatial resolution of the reference image differs from the spatial resolution of the current image. When motion vector scaling is enabled, the motion vector is scaled based on both the POC difference and the spatial resolution ratio between the current picture and the reference picture.

[110] In the same or another embodiment, the motion vector can be scaled based on the AUC difference instead of the POC difference for temporal motion vector prediction, especially when poc_cycle_au has a non-uniform value (when vps_contant_poc_cycle_per_au=0). Otherwise (when vps_contant_poc_cycle_per_au=1), motion vector scaling based on AUC difference may be identical to motion vector scaling based on POC difference.

[111] In the same or different embodiment, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU (with the same AUC value) with the current image is not scaled based on the AUC difference and is used to predict the motion vector without scaling or scaling based on the ratio of spatial resolution between the current image and the reference image.

[112] In the same and other embodiments, the AUC value is used to identify the AU boundary and is used for a hypothetical reference decoder (HRD) operation that requires input and output synchronization with AU granularity. In most cases, the decoded picture with the highest level in the AU can be output for display. The AUC value and the layer_id value may be used to identify the output image.

[113] In an embodiment, an image may consist of one or more sub-images. Each sub-image may cover a local area or an entire image area. The area supported by a sub-image may or may not overlap with the area supported by another sub-image. An area composed of one or more sub-images may or may not cover the entire image area. If the image consists of a sub-image, the area supported by the sub-image is identical to the area supported by the image.

[114] In the same embodiment, the sub-image may be encoded by a coding method similar to the coding method used for the encoded image. A sub-image may be independently encoded, or may be encoded depending on another sub-image or encoded image. A sub-image may or may not have any parsing dependency on another sub-image or encoded image.

[115] In the same embodiment, the encoded sub-image may be contained in one or more layers. The encoded sub-image in the layer may have a different spatial resolution. The original sub-image may be spatially resampled (upsampled or downsampled), encoded with different spatial resolutions, and contained in the bitstream corresponding to the layer.

[116] In the same or another embodiment, a sub-image with (W, H), where W denotes the width of the sub-image and H denotes the height of the sub-image, respectively, can be encoded and contained in the encoded bitstream corresponding to layer 0, while the sub-image upsampled (or downsampled) from the sub-image with the original spatial resolution, with (W*S _w,k , H*S _h,k ), may be encoded and contained in the encoded bitstream corresponding to layer k, where S _w,k , S _h,k indicate horizontal and vertical resampling factors. If the values of S _w,k , S _h,k are greater than 1, oversampling is equal to upsampling. Whereas if the values of S _w,k , S _h,k are less than 1, oversampling is equal to downsampling.

[117] In the same or another embodiment, an encoded sub-image in a layer may have a different visual quality than an encoded sub-image in another layer in the same sub-image or another sub-image. For example, sub-image i in layer n is encoded with a quantization parameter Q _i,n , while sub-image j in layer m is encoded with a quantization parameter Q _j,m .

[118] In the same or another embodiment, an encoded sub-image in a layer can be decoded independently, without any parsing or decoding dependency, from an encoded sub-image in another layer of the same local area. A sub-image layer that can be independently decoded without referring to another sub-image layer of the same local area is an independent sub-image layer. An encoded sub-image in an independent sub-image layer may or may not have a decoding or parsing dependency on a previously encoded sub-image in the same sub-image layer, but an encoded sub-image may not have any dependency on an encoded image in another sub-image layer.

[119] In the same or another embodiment, an encoded sub-image in a layer can be decoded dependently, with any parsing or decoding dependency, from an encoded sub-image in another layer of the same local area. A sub-image layer that can be dependently decoded with reference to another sub-image layer of the same local area is a dependent sub-image layer. An encoded sub-image in a dependent sub-image may refer to an encoded sub-image belonging to the same sub-image, a previously encoded sub-image in the same sub-image layer, or both reference sub-images.

[120] In the same or another embodiment, the encoded sub-image consists of one or more independent sub-image layers and one or more dependent sub-image layers. However, at least one independent sub-picture may be present for an encoded sub-picture. An independent sub-image layer may have a layer identifier (layer_id) value, which may be present in a NAL unit header or other high-level syntax structure, equal to 0. A sub-image layer with layer_id equal to 0 is the base sub-image layer.

[121] In the same or another embodiment, an image may be composed of one or more foreground sub-images and one background sub-image. The area supported by the background sub-image may be equal to the image area. The area supported by the foreground sub-image may overlap with the area supported by the background sub-image. The background sub-image may be a base sub-image layer, while the foreground sub-image may be a non-base (enhancement) sub-image layer. One or more non-base sub-image layers may refer to the same base layer for decoding. Each non-base subimage layer with layer_id equal to a may refer to a non-base subimage layer with layer_id equal to b, where a is greater than b.

[122] In the same or another embodiment, the image may consist of one or more foreground sub-images with or without a background sub-image. Each sub-image may have its own base sub-image layer and one or more non-base (enhancement) layers. Each base subimage layer may be referenced by one or more non-base subimage layers. Each non-base subimage layer with layer_id equal to a may refer to a non-base subimage layer with layer_id equal to b, where a is greater than b.

[123] In the same or another embodiment, the image may consist of one or more foreground sub-images with or without a background sub-image. Each encoded sub-image in a (base or non-base) sub-image layer may be referenced by one or more non-base-layer sub-images that belong to the same sub-image and one or more non-base-layer sub-images that do not belong to the same sub-image .

[124] In the same or another embodiment, the image may consist of one or more foreground sub-images with or without a background sub-image. The sub-image in layer a can be further divided into a plurality of sub-images in the same layer. One or more encoded sub-images in layer b may refer to a split sub-image in layer a.

[125] In the same or another embodiment, a coded video sequence (CVS) may be a group of coded pictures. The CVS may be composed of one or more coded sub-picture sequences (CSPS), where the CSPS may be a group of coded sub-pictures covering the same local area of an image. The CSPS may have the same or different temporal resolution as the encoded video sequence.

[126] In the same or another embodiment, the CSPS may be encoded and contained in one or more layers. The CSPS may be composed of one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS may recover a sequence of sub-images corresponding to the same local area.

[127] In the same or another embodiment, the number of CSPS layers corresponding to a CSPS may be identical or different from the number of CSPS layers corresponding to another CSPS.

[128] In the same or another embodiment, a CSPS layer may have a different temporal resolution (eg, frame rate) than another CSPS layer. The original (uncompressed) sub-image sequence may be temporally resampled (upsampled or downsampled), encoded with different temporal resolution parameters, and contained in the bitstream corresponding to the layer.

[129] In the same or another embodiment, a sub-image sequence with frame rate F may be encoded and contained in the coded bitstream corresponding to layer 0, while a temporally upsampled (or downsampled) sub-image sequence from the original sub-image sequence with F *S _t,k may be encoded and contained in the encoded bitstream corresponding to layer k, where S _t,k indicates the temporal sampling factor for layer k. If the value of S _t,k is greater than 1, the temporal resampling process is equal to frame rate upconversion. Whereas, if the value of S _t,k is less than 1, the temporal resampling process is equal to frame rate downconversion.

[130] In the same or another embodiment, when the sub-image with CSPS layer a is a sub-image reference with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b, the decoded pixels in the layer CSPS a are resampled and used for reference. The resampling process may require upsampling or downsampling filtering.

[131] FIG. 9 shows an example of syntax tables for signaling a vps_poc_cycle_au syntax element in a VPS (or SPS) that specifies the poc_cycle_au used for all pictures/slices in a coded video sequence, and a slice_poc_cycle_au syntax element that specifies the current slice's poc_cycle_au in the slice header. If the POC value increases evenly for each AU, vps_contant_poc_cycle_per_au in the VPS is set to 1 and vps_poc_cycle_au is signaled in the VPS. In this case slice_poc_cycle_au is not explicitly signaled and the AUC value for each AU is calculated by dividing the POC value by vps_poc_cycle_au. If the POC value does not increase uniformly per AU, vps_contant_poc_cycle_per_au in the VPS is set to 0. In this case, vps_access_unit_cnt is not signaled, while slice_access_unit_cnt is signaled in the slice header for each slice or picture. Each slice or image can have a different slice_access_unit_cnt value. The AUC value for each AU is calculated by dividing the POC value by slice_poc_cycle_au. In FIG. 10 is a flowchart illustrating the corresponding workflow.

[132] In the same or other embodiments, even though the POC value of an image, slice, or tile may be different, the image, slice, or tile corresponding to AUs with the same AUC value may be associated with the same decoding or output time. . Therefore, without any dependency on interparsing and/or decoding between images, slices or tiles in the same AU, all or a subset of the images, slices or tiles associated with the same AU can be decoded in parallel and can be taken out at the same time.

[133] In the same or other embodiments, even though the POC value of an image, slice, or tile may be different, the image, slice, or tile corresponding to AUs with the same AUC value may be associated with the same layout/display time. . When the link time is contained in the container format, even if the images correspond to different AUs, if the images have the same link time, the images may be displayed at the same point in time.

[134] In the same or other embodiments, each image, slice, or tile may have the same temporal identifier (temporal_id) in the same AU. All or a subset of images, slices or tiles corresponding to a point in time may be associated with the same temporal sublayer. In the same or different embodiments, each image, slice or tile may have the same or a different spatial layer identifier (layer_id) in the same AU. All or a subset of images, slices or tiles corresponding to a point in time may be associated with the same or a different spatial layer.

[135] FIG. 11 shows an exemplary video stream including a background CSPS video with layer_id equal to 0 and a plurality of foreground CSPS layers. Although the encoded sub-image may be composed of one or more CSPS layers, the background region that does not belong to any foreground CSPS layer may be composed of the base layer. The base layer may contain the background and foreground regions, while the enhancement layer CSPS contains the foreground region. The CSPS enhancement layer may have better visual quality than the base layer in the same area. The enhancement layer CSPS may refer to the reconstructed pixels and motion vectors of the base layer corresponding to the same area.

[136] In the same or another embodiment, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers corresponding to each sub-image are contained in a separate track in the video file.

[137] In the same or another embodiment, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers with the same layer_id are contained in a separate track. In this example, the track corresponding to layer k includes only the CSPS layers corresponding to layer k.

[138] In the same or different embodiment, each CSPS layer of each sub-image is stored in a separate track. Each track may or may not have a parsing or decoding dependency on one or more other tracks.

[139] In the same or another embodiment, each track may contain bitstreams corresponding to layers i to j of CSPS layers of all or a subset of sub-images, where 0<r=j=k, k is the layer of the highest CSPS layer.

[140] In the same or another embodiment, an image consists of one or more associated media data, including a depth map, an alpha map, 3D geometry data, an occupancy map, and so on. Such associated synchronized media data may be divided into one or multiple sub-data streams, each corresponding to one sub-picture.

[141] In the same or another embodiment, FIG. 12 shows an example of a video conference based on the layered sub-image method. The video stream contains one base layer video bitstream corresponding to the background image and one or more enhancement layer video bitstreams corresponding to the foreground sub-images. Each video bitstream of the enhancement layer corresponds to a CSPS layer. The default display shows the image corresponding to the base layer. It contains a picture of one or more users in a picture (PIP). When a particular user is selected by the client control, the enhancement layer CSPS corresponding to the selected user is decoded and displayed with enhanced quality or spatial resolution. In FIG. 13 shows a block diagram of operation.

[142] In the same or a different embodiment, an intermediate network unit (such as a router) may select a subset of layers to send to the user depending on its bandwidth. Image/sub-image organization can be used for bandwidth adaptation. For example, if the user does not have bandwidth, the router splits layers or selects some sub-images because of their importance or based on the settings used, and this can be done dynamically to adapt to the bandwidth.

[143] FIG. 14 shows an example of using 360-degree video. When a spherical 360-degree image is projected onto a flat image, the projection image 360 may be divided into a plurality of sub-images as a base layer. A particular sub-image enhancement layer may be encoded and transmitted to the client. The decoder may decode both the base layer including all sub-images and the enhancement layer of the selected sub-image. When the current viewport is identical to the selected sub-image, the displayed image may be of higher quality with the decoded sub-image with the enhancement layer. Otherwise, the base layer decoded image may be displayed with low quality.

[144] In the same or another embodiment, any layout information for display may be present in the file as additional information (eg, SEI messages or metadata). One or more decoded sub-images may be moved and displayed depending on the transmitted layout information. The layout information may be transmitted by a streaming server or a broadcaster, or may be retrieved by a network entity or a cloud server, or may be determined by user customization.

[145] In an embodiment, when the input image is divided into one or more (rectangular) sub-regions, each sub-region may be encoded as an independent layer. Each independent layer corresponding to a local area can have a unique layer_id value. For each independent layer, information about the size and location of the sub-image can be transmitted. For example, image size (width, height), information about the offset of the upper left corner (х_offset, y_offset). In FIG. 15 shows an example of a split image layout, size and position information of its sub-images, and its corresponding image prediction structure. Layout information, including sub-image size(s) and sub-image position(s), may be conveyed in a high-level syntax structure such as a parameter set(s), a slice or tile group header, or an SEI message.

[146] In the same embodiment, each sub-image corresponding to an independent layer may have its own unique POC value within the AU. When a reference image among the images stored in the DPB is specified using the syntax element(s) in the RPS or RPL structure, the POC value(s) of each sub-image corresponding to the layer may be used.

[147] In the same or another embodiment, to indicate the structure of the (interlayer) prediction, layer_id may not be used, and a POC (delta) value may be used.

[148] In the same embodiment, a sub-image with a POC value of N corresponding to a layer (or a local area) may or may not be used as a reference image of a sub-image with a POC value of N+K corresponding to the same layer (or the same same local area) for motion-compensated prediction. In most cases, the value of the K number may be equal to the maximum number of (independent) layers, which may be identical to the number of subregions.

[149] In the same or another embodiment, FIG. 16 shows an extended example of FIG. 15. When the input image is divided into multiple (eg, four) sub-regions, each local region may be encoded with one or more layers. In this case, the number of independent layers may be equal to the number of sub-regions, and one or more layers may correspond to a sub-region. Thus, each sub-region may be encoded with one or more independent layers and zero or more dependent layers.

[150] In the same embodiment shown in FIG. 16, the input image may be divided into four sub-regions. The upper right sub-region may be encoded as two layers, which are layer 1 and layer 4, while the lower right sub-region may be encoded as two layers, which are layer 3 and layer 5. In this case, layer 4 may refer to layer 1 for motion compensated prediction, while layer 5 may refer to layer 3 for motion compensation.

[151] In the same or another embodiment, in-loop filtering (eg, deblocking filtering, adaptive in-loop filtering, reshaping, two-way filtering, or any deep learning-based filtering) across the layer boundary can be (optionally) disabled.

[152] In the same or another embodiment, motion-compensated prediction or intra-block copying across a layer boundary can be (optionally) disabled.

[153] In the same or another embodiment, boundary padding for motion-compensated prediction or in-loop filtering at a sub-image boundary may be optionally processed. A flag indicating whether edge padding is handled or not may be conveyed in a high-level syntax structure such as a parameter set(s) (VPS, SPS, PPS, or APS), a slice or tile group header, or an SEI message.

[154] In the same or another embodiment, layout information of the sub-area(s) (or sub-image(s)) may be transmitted to the VPS or SPS. In FIG. 17 shows an example of syntax elements in VPS and SPS. In this example, the vps_sub_picture_dividing_flag is passed to the VPS. The flag may indicate whether the input(s) image(s) is divided into multiple sub-areas or not. When the value of vps_sub_picture_dividing_flag is 0, the input picture(s) in the encoded video sequence(s) corresponding to the current VPS cannot be divided into multiple sub-regions. In this case, the size of the input image may be equal to the size of the encoded image (pic_width_in_luma_samples, pic_height_in_luma_samples) that is signaled in the SPS. When the value of vps_sub_picture_dividing_flag is 1, the input image(s) can be divided into a plurality of sub-regions. In this case, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled to the VPS. The values vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples can be equal to the width and height of the input image(s), respectively.

[155] In the same embodiment, the values vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples cannot be used for decoding, but can be used for layout and display.

[156] In the same embodiment, when the value of vps_sub_picture_dividing_flag is equal to 1, the syntax elements pic_offset_x and pic_offset_y can be transmitted in the SPS corresponding to the particular layer(s). In this case, the size of the encoded image (pic_width_in_luma_samples, pic_height_in_luma_samples) transmitted in the SPS may be equal to the width and height of the sub-region corresponding to a particular layer. Also, the position (pic_offset_x, pic_offset_y) of the upper left corner of the sub-area may be transmitted to the SPS.

[157] In the same embodiment, the position information (pic_offset_x, pic_offset_y) of the upper left corner of the sub-area cannot be used for decoding, but can be used for layout and display.

[158] In the same or another embodiment, layout information (size and position) of all or a subset of sub-areas of the input image(s), dependency information between layer(s) may be signaled in a parameter set or SEI message. In FIG. 18 shows an example of syntax elements for specifying information about the layout of subregions, dependencies between layers, and relationships between a subregion and one or more layers. In this example, the num_sub_region syntax element indicates the number of (rectangular) sub-regions in the current encoded video sequence; the num_layers syntax element specifies the number of layers in the current encoded video sequence. The num_layers value can be equal to or greater than the num_sub_region value. When any sub-region is encoded as a single layer, the value of num_layers may be equal to the value of num_sub_region. When one or more sub-regions are encoded as multiple layers, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag [i] [j] indicates a dependency from the jth level to the i-th level; num_layers_for_region [i] indicates the number of layers associated with the i-th sub-region; sub_region_layer_id [i] [j] indicates the layer_id of the j-th layer associated with the i-th sub-region. The values sub_region_offset_x[i] and sub_region_offset_y[i] indicate, respectively, the horizontal and vertical position of the upper left corner of the i-th sub-region. The sub_region_width [i] and sub_region_height [i] values indicate the width and height of the i-th subregion, respectively.

[159] In one embodiment, one or more syntax elements that define a set of output layers to specify one or more layers to be output with or without profile tier information may be conveyed in a high-level syntax structure, such as VPS, DPS, SPS, PPS APS or SEI message. Referring to FIG. 19, a num_output_layer_sets syntax element indicating the number of output layer sets (OLS) in a coded video sequence related to the VPS may be transmitted to the VPS. For each set of output layers, the output_layer_flag can be passed as many as there are output layers.

[160] In the same embodiment, output_layer_flag [i] equal to 1 specifies that the i-th layer is output; vps_output_layer_flag [i] equal to 0 specifies that the i-th layer is not output.

[161] In the same or a different embodiment, one or more syntax elements that specify profile tier information for each set of output layers may be conveyed in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message. Still referring to FIG. 19, the num_profile_tile_level syntax element indicating the amount of profile tier information on the OLS in the encoded video sequence related to the VPS may be signaled to the VPS. For each set of output layers, a set of syntax elements for profile tier information or an index indicating specific profile tier information among entries in the profile tier information may be signaled as many as there are output layers.

[162] In the same embodiment, profile_tier_level_idx[i][j] specifies an index in the list of profile_tier_level() syntax structures in the VPS of the profile_tier_level() syntax structure that applies to the j-th layer of the i-th OLS.

[163] In the same or different embodiment, as shown in FIG. 20, the syntax elements num_profile_tile_level and/or num_output_layer_sets may be signaled when the number of maximum layers is greater than 1 (vps_max_layers_minus 1>0).

[164] In the same or different embodiment, as shown in FIG. 20, the vps_output_layers_mode[i] syntax element indicating the output layer signaling mode for the i-th set of output layers may be present in the VPS.

[165] In the same embodiment, vps_output_layers_mode[i] equal to 0 indicates that only the topmost layer is output with the i-th set of output layers; vps_output_layer_mode [i] equal to 1 indicates that all layers are output with the i-th set of output layers; vps_output_laye_mode[i] equal to 2 indicates that the output layers are layers with vps_output_layer_flag[i][j] equal to 1, with the i-th set of output layers. Other values may be reserved.

[166] In the same embodiment, output_layer_flag[i][j] may or may not be signaled depending on the value of vps_output_layers_mode[i] for the i-th set of output layers.

[167] In the same or different embodiment, as shown in FIG. 20, the vps_pt1_signal_flag[i] flag may be present for the i-th set of output layers. Depending on the value of vps_pt1_signal_flag[i], the profile tier information for the i-th set of output layers may or may not be signaled.

[168] In the same or different embodiment, as shown in FIG. 21, the subpicture number, max_subpics_minus1, in the current CVS may be signaled in a high-level syntax structure, such as in a VPS, DPS, SPS, PPS, APS, or SEI message.

[169] In the same embodiment, as shown in FIG. 21, the sub-picture ID, sub_pic_id[i], for the i-th sub-picture may be signaled when the number of sub-pictures is greater than 1 (max_subpics_minus1>0).

[170] In the same or a different embodiment, one or more syntax elements indicating a sub-image identifier belonging to each layer of each set of output layers may be signaled to the VPS. Referring to FIG. 22, sub_pic_id_layer[i][j][k], which indicates the k-th sub-image present in the j-th layer of the i-th set of output layers. With this information, the decoder can determine which sub-image can be decoded and output for each layer of a particular set of output layers.

[171] In an embodiment, a picture header (PH) is a syntactic structure containing syntax elements that apply to all slices of an encoded picture. A picture unit (PU) is a set of NAL units that are related to each other according to a given classification rule, are sequential in decoding order, and contain exactly one encoded picture. The PU may contain a picture header (PH) and one or more VCL NAL units constituting the encoded picture.

[172] In an embodiment, the SPS (RBSP) may be available to the decoding process before being referenced, included in at least one AU with TemporalId equal to 0, or provided through external means.

[173] In an embodiment, the SPS (RBSP) may be available to the decoding process before it is referenced, it will be included in at least one AU with TemporalId equal to 0 in a CVS that contains one or more PPS, related to the SPS, or provided through external means.

[174] In an embodiment, an SPS (RBSP) may be available to a decoding process before it is referenced by one or more PPSs, it will be included in at least one PU with a nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL units that refers to an SPS NAL unit in a CVS that contains one or more SPS-related PPSs or is provided through external means.

[175] In an embodiment, an SPS (RBSP) may be available to the decoding process before it is referenced by one or more PPSs, it will be included in at least one PU with TemporalId equal to 0 and nuh_layer_id equal to the lowest value The nuh_layer_id of the PPS NAL units that belong to the SPS NAL unit or are provided through external means.

[176] In an embodiment, an SPS (RBSP) may be available to the decoding process before it is referenced by one or more PPSs, it will be included in at least one PU with TemporalId equal to 0 and nuh_layer_id equal to the lowest value of nuh_layer_id PPS NAL units that refer to an SPS NAL unit in a CVS that contains one or more SPS related PPS or is provided through external means.

[177] In the same or a different embodiment, pps_seq_parameter_set_id indicates the sps_seq_parameter_set_id value for the SPS that is being referenced. The value of pps_seq_parameter_set_id may be the same in all PPS referenced by encoded pictures in CLVS.

[178] In the same or another embodiment, all SPS NAL units with a particular sps_seq_parameter_set_id value in CVS may have the same content.

[179] In the same or another embodiment, regardless of the nuh_layer_id values, the SPS NAL units may share the same sps_seq_parameter_set_id value space.

[180] In the same or another embodiment, the nuh_layer_id value of the SPS NAL unit may be equal to the smallest nuh_layer_id value of the PPS NAL units that belong to the SPS NAL unit.

[181] In an embodiment, when an SPS with nuh_layer_id equal to m is referenced by one or more PPS with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or ( direct or indirect) reference layer of a layer with nuh_layer_id equal to m.

[182] In an embodiment, the PPS (RBSP) must be available to the decoding process before being referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit, or provided through external means.

[183] In an embodiment, the PPS (RBSP) may be available to the decoding process before it is referenced, it will be included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS that contains one or more PH (or coded slice NAL units) related to the PPS or provided through external means.

[184] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it will be included in at least one PU with nuh_layer_id equal to the lowest to the nuh_layer_id value of the coded slice NAL units that refer to the PPS NAL unit in the CVS, which contains one or more PHs (or coded slice NAL units) related to the PPS, or is provided through external means.

[185] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it will be included in at least one PU with TemporalId equal to TemporalId a PPS NAL unit and a nuh_layer_id equal to the lowest value of the nuh_layer_id coded slice NAL units that refer to a PPS NAL unit in a CVS that contains one or more PPS-specific PHs (or coded slice NAL units) or is provided through external means.

[186] In the same or a different embodiment, the ph_pic_parameter_set_id in the PH indicates the value of pps_pic_parameter_set_id for the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPS referenced by encoded pictures in CLVS.

[187] In the same or another embodiment, all PPS NAL units with a particular pps_pic_parameter_set_id value in a PU may have the same content.

[188] In the same or another embodiment, regardless of the nuh_layer_id values, PPS NAL units may share the same pps_pic_parameter_set_id value space.

[189] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the smallest nuh_layer_id value of the coded slice NAL units that refer to the PPS NAL unit.

[190] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more encoded slice NAL units with nuh_laye_ id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or (directly or indirectly) the reference layer of a layer with nuh_layer_id equal to m.

[191] In an embodiment, the PPS (RBSP) must be available to the decoding process before being referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit, or provided through external means.

[192] In an embodiment, the PPS (RBSP) may be available to the decoding process before it is referenced, it will be included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS that contains one or more PH (or coded slice NAL units) related to the PPS or provided through external means.

[193] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it will be included in at least one PU with nuh_layer_id equal to the lowest to the nuh_layer_id value of the coded slice NAL units that refer to the PPS NAL unit in the CVS, which contains one or more PHs (or coded slice NAL units) related to the PPS, or is provided through external means.

[194] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it will be included in at least one PU with TemporalId equal to TemporalId a PPS NAL unit and a nuh_layer_id equal to the lowest value of the nuh_layer_id coded slice NAL units that refer to a PPS NAL unit in a CVS that contains one or more PPS-specific PHs (or coded slice NAL units) or is provided through external means.

[195] In the same or a different embodiment, the ph_pic_parameter_set_id in the PH indicates the value of pps_pic_parameter_set_id for the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPS referenced by encoded pictures in CLVS.

[196] In the same or another embodiment, all PPS NAL units with a particular pps_pic_parameter_set_id value in a PU may have the same content.

[197] In the same or another embodiment, regardless of the nuh_layer_id values, PPS NAL units may share the same pps_pic_parameter_set_id value space.

[198] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the smallest nuh_layer_id value of the coded slice NAL units that refer to the PPS NAL unit.

[199] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more encoded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n , or (directly or indirectly) the reference layer of the layer with nuh_layer_id equal to m.

[200] In an embodiment, when the no_temporal_sublayer_switching_flag flag is signaled to the DPS, VPS, or SPS, the TemporalId value of the PPS referring to the parameter set containing the flag equal to 1 may be 0, while the TemporalId value of the PPS referring to the parameter set A containing a flag of 1 can be equal to or greater than the TemporalId value of the option set.

[201] In an embodiment, each PPS (RBSP) may be available to the decoding process before it is referenced, it will be included in at least one AU with a TemporalId less than or equal to the TemporalId of the encoded slice NAL unit (or NAL unit PH) that references it or is provided through external means. When a PPS NAL unit is included in an AU before an AU containing a PPS-related coded slice NAL unit, a VCL NAL unit providing upper layer temporal switching, or a VCL NAL unit with nal_unit_type equal to STSA NUT, indicating that the picture in the VCL NAL unit may be a Stepwise Temporal Sublayer Access (STSA) picture, may not be present after the PPS NAL unit and before the coded slice NAL unit referring to the APS.

[202] In the same or another embodiment, a PPS NAL unit and a coded slice NAL unit (and its PH NAL unit) referring to a PPS may be included in the same AU.

[203] In the same or a different embodiment, the PPS NAL unit and the STS A NAL unit may be included in the same AU that precedes the coded slice NAL unit (and its PH NAL unit) referring to the PPS.

[204] In the same or another embodiment, an STSA NAL unit, a PPS NAL unit, and a coded slice NAL unit (and its PH NAL unit) referring to a PPS may be included in the same AU.

[205] In the same embodiment, the TemporalId value of the VCL NAL unit containing the PPS may be equal to the TemporalId value of the previous STSA NAL unit.

[206] In the same embodiment, the value of the picture order counter (POC) of the NAL PPS unit may be equal to or greater than the value of the POC of the NAL STSA unit.

[207] In the same embodiment, the value of the picture order counter (POC) of the coded slice or NAL unit PH that refers to the PPS NAL unit may be equal to or greater than the POC value of said PPS NAL unit.

[208] In an embodiment, the value of sps_max_sublayers_minus1 must be the same for all layers in the encoded video sequence because all VCL NAL units in the AU must have the same TemporalId value. The value of sps_max_sublayers_minus1 must be the same across all SPS referenced by encoded images in CVS.

[209] In an embodiment, the chroma_format_idc value of the SPS referenced by one or more coded pictures in layer A must be equal to the chroma_format_idc value of the SPS referenced by one or more coded pictures in layer B, where layer A is a direct reference layer. layer B. This is because any encoded picture must have the same chroma_format_idc value as its reference picture. The chroma_format_idc value of an SPS referenced by one or more coded images in layer A must be equal to the value of chroma_format_idc in the SPS referenced by one or more coded images in the forward reference layer of layer A in CVS.

[210] In an embodiment, the values of subpics_present_flag and sps_subpic_id_present_flag for an SPS referenced by one or more coded pictures in layer A should be equal to the values of subpics_present_flag and sps_subpic_id_present_flag in an SPS referenced by one or more coded pictures in layer B, where layer A is the direct reference layer of layer B. This is because the sub-image layout must be aligned or linked between layers. Otherwise, a sub-image with many layers may not be extracted correctly. The values of subpics_present_flag and sps_subpic_id_present_flag for the SPS referenced by one or more coded pictures in layer A shall be equal to the values of subpics_present_flag and sps_subpic_id_present_flag in the SPS referenced by one or more coded pictures in the forward reference layer of layer A in CVS.

[211] In an embodiment, when an STSA image in layer A is referenced by an image in a forward reference layer of layer A in the same AU, the STSA-related image must be an STSA image. Otherwise, sublayer timing cannot be synchronized between layers. When a STSA NAL unit in stratum A is referenced by a VCL NAL unit in stratum A's forward reference layer in the same AU, the nal_unit_type value of the VCL NAL unit referring to the STSA NAL unit shall be equal to STSA_NUT.

[212] In an embodiment, when a RASL image in layer A is referenced by an image in a forward reference layer of layer A in the same AU, the RASL-related image must be a RASL image. Otherwise, the image cannot be decoded correctly. When a RASL NAL unit in stratum A is referenced by a VCL NAL unit in stratum A's forward reference layer in the same AU, the nal_unit_type value of the VCL NAL unit referring to the RASL NAL unit shall be equal to RASL_NUT.

[213] While this disclosure describes several exemplary embodiments, there are variations, permutations, and various replacement equivalents that fall within the scope of the disclosure. Thus, it will be appreciated that those skilled in the art will be able to develop numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the disclosure and thus fall within its spirit and scope.

Claims (43)

1. A method for signaling sets of output layers in an encoded video stream, performed by a processor, comprising: - receiving video data having multiple layers; - identifying one or more first syntax elements indicating one or more sets of output layers corresponding to output layers from a plurality of layers of the received video data; And - decoding and displaying one or more output layers corresponding to the specified sets of output layers, wherein one or more syntax elements are signaled in a high-level syntax structure, wherein the high-level syntax structure comprises one of the following: a video parameter set, a dependency parameter set, a sequence parameter set, an image parameter set, an adaptation parameter set, and reporting additional enhancement information, wherein the sequence parameter set is available for decoding before the sequence parameter set is referenced by one or more of the picture parameter sets. 2. The method according to p. 1, further comprising: - an indication of the number of one or more sets of output layers; and signaling the number of output layers by setting an output layer flag. 3. The method of claim 1, further comprising identifying one or more second syntax elements indicating profile tier level information for each of the one or more output layer sets. 4. The method of claim 3, wherein the one or more second syntax elements are signaled in a high-level syntax structure. 5. The method of claim 4, wherein the high-level syntax structure comprises one of the following: a video parameter set, a dependency parameter set, a sequence parameter set, an image parameter set, an adaptation parameter set, and reporting additional enhancement information. 6. The method of claim 3, further comprising: - specifying the amount of profile tier information for each of the one or more sets of output layers; And - signaling, for each of the one or more sets of output layers, a set of syntax elements corresponding to the profile tier information, or an index indicating a record of the profile tier information from the profile tier information. 7. The method of claim 1, further comprising signaling one or more third syntax elements indicating a sub-image identifier corresponding to each output layer associated with the one or more output layer sets. 8. The method of claim 7, wherein the sub-image identifier is signaled in a high-level syntax structure comprising one of the following: a video parameter set, a dependency parameter set, a sequence parameter set, an image parameter set, an adaptation parameter set, and reporting additional enhancement information. 9. A computer system for signaling output layer sets in encoded video data, the computer system comprising: - one or more machine-readable non-transitory storage media configured to store computer program code; And - one or more computer processors configured to access said computer program code and operate in accordance with the instructions of said computer program code, said computer program code including: - a reception code configured to cause one or more computer processors to receive video data having multiple layers; - an identification code configured to cause one or more computer processors to identify one or more first syntax elements indicating one or more sets of output layers corresponding to output layers from a plurality of layers of received video data; And - decoding and display code configured to cause one or more computer processors to decode and display one or more output layers corresponding to specified sets of output layers, wherein one or more syntax elements are signaled in a high-level syntax structure, wherein the high-level syntax structure comprises one of the following: a video parameter set, a dependency parameter set, a sequence parameter set, an image parameter set, an adaptation parameter set, and reporting additional enhancement information, wherein the sequence parameter set is available for decoding before the sequence parameter set is referenced by one or more of the picture parameter sets. 10. The computer system according to claim 9, further comprising: - an indication code configured to cause one or more computer processors to indicate the number of one or more sets of output layers; And - a signaling code configured to cause one or more computer processors to signal the number of output layers by setting an output layer flag. 11. The computer system of claim 9, further comprising an identification code configured to cause one or more computer processors to identify one or more second syntax elements indicating profile tier information for each of the one or more output layer sets. 12. The computer system of claim 11, wherein the one or more second syntax elements are signaled in a high-level syntax structure. 13. The computer system of claim 12, wherein the high-level syntax structure comprises one of the following: a video parameter set, a dependency parameter set, a sequence parameter set, an image parameter set, an adaptation parameter set, and reporting additional enhancement information. 14. The computer system according to claim 11, further comprising: - an indication code configured to cause one or more computer processors to indicate the amount of profile tier information for each of the one or more sets of output layers; And - a signaling code configured to cause one or more computer processors to signal, for each of the one or more output layer sets, a set of syntax elements corresponding to the profile tier information, or an index indicating a profile tier information entry from the profile tier information. 15. The computer system of claim 9, further comprising a signaling code configured to cause one or more computer processors to signal one or more third syntax elements indicating a sub-image identifier corresponding to each output layer associated with one or more output layer sets. 16. A non-transitory computer-readable medium that stores a computer program for signaling output layer sets in an encoded video stream, the computer program configured to cause one or more computer processors to: - receive video data having multiple layers; - identify one or more first syntax elements indicating one or more sets of output layers corresponding to output layers from a plurality of layers of received video data; And - decode and display one or more output layers corresponding to the specified sets of output layers, wherein one or more syntax elements are signaled in a high-level syntax structure, wherein the high-level syntax structure comprises one of the following: a video parameter set, a dependency parameter set, a sequence parameter set, an image parameter set, an adaptation parameter set, and reporting additional enhancement information, wherein the sequence parameter set is available for decoding before the sequence parameter set is referenced by one or more of the picture parameter sets.

Images