RU2804274C1 - Image output method with a set of output layers - Google Patents

Abstract

FIELD: video data processing.

SUBSTANCE: video encoding and decoding, and more specifically, image output with a set of output layers. Systems and methods for encoding and decoding are proposed. The method includes receiving an encoded video stream including video data divided into multiple layers; and decoding the video data of the encoded video stream, the decoding including: determining whether a first image of the encoded video stream is to be output based on a syntax element that indicates a signaling mode of the output layer set.

EFFECT: resampling of the reference image and the availability of adaptive resolution encoding signals for parts of the encoded image.

15 cl, 31 dwg

Description

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 62/966,127, filed January 27, 2020, and U.S. Patent Application No. 17/063937, filed October 6, 2020, which are incorporated herein in their entirety.

TECHNICAL FIELD

[0002] Embodiments of the present invention relate to video encoding and decoding, and more specifically, to outputting an image with a set of output layers.

BACKGROUND OF THE ART

[0003] Video encoding and decoding using inter-frame prediction with motion compensation has been used previously. Uncompressed digital video may include a series of images, with each image having a spatial size, for example, 1920x1080 luminance samples and associated chrominance samples. A series of images may have a fixed or variable image rate (colloquially also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 luminance sample resolution at 60 Hz frame rate) requires a bandwidth of close to 1.5 Gbps. An hour of such video requires more than 600 GB of storage space.

[0004] One purpose of video encoding and decoding may be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or memory space requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, or a combination of both, can be used. Lossless compression refers to technologies in which an exact copy of the original signal can be reconstructed from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals may be small enough to make the reconstructed signal suitable for the intended application. In the case of video, lossy compression is widely used. The acceptable degree of distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of broadcast television applications. The achievable compression ratio may reflect the following: higher tolerable/acceptable distortion may produce a higher compression ratio.

[0005] The video encoder and decoder may use techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding, some of which will be presented below.

[0006] Previously, video encoders and decoders tended to operate at a given picture size, which in most cases was defined and remained constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2, system designs were used to change horizontal resolution (and thus image size) depending on factors such as scene activity, but only for I images, hence typically GOP. Resampling of reference images to accommodate different resolutions in CVS has been used, for example, in ITU-T Rec. H.263 Appendix P However, here the image size is not changed, only the reference images are resampled, so that only parts of the image canvas (in the case of downsampling) or only parts of the captured scene (in the case of upsampling) can be used. In addition, H.263 Annex Q allows resampling of an individual macroblock by a factor of two (in each dimension), up or down. Again, the image size remains the same. The macroblock size is fixed in H.263 and therefore does not need to be signaled.

[0007] Image resizing in predicted images has become more common in modern video coding. For example, VP9 allows you to resample a reference image and change the resolution of the entire image. Likewise, some of the proposals made regarding VVC (including, for example, Hendry, et al., "On Adaptive Resolution Changes (ARC) for VVC," Joint Video Team JVET-M0135-vl, January 9-18, 2019, included herein in its entirety) allow entire reference images to be resampled at a different higher or lower resolution. Such a document proposes to encode different resolution options in a sequence parameter set and refer to them using syntax elements for each image in the image parameter set.

DISCLOSURE OF THE INVENTION

[0008] Compressed region aggregation, or extracting multiple semantically independent parts of an image into a single video image, has received some attention recently. Particularly in the context of, for example, 360 encoding or certain surveillance applications, multiple semantically independent source images (for example, the surface of six cubes of a 360 cube-projected scene, or individual camera inputs in the case of a multi-camera surveillance setup) may require separate adaptive resolution settings to cope with different activity of each scene at a given time. In other words, coders at a given time may choose to use different resampling coefficients for different semantically independent images that make up the entire 360-degree or surveillance scene. When independent images are combined into a single image, it becomes necessary to resample the reference image and make adaptive resolution encoding signals available for portions of the encoded image.

[0009] Embodiments of the present invention may satisfy the above need and/or solve other problems.

[0010] According to one or more embodiments, a method is provided that is executed by at least one processor. The method includes: receiving an encoded video stream including video data divided into a plurality of layers; and decoding video data of the encoded video stream, wherein the decoding includes: determining whether a first image of the encoded video stream is to be output based on a syntax element that indicates a signaling mode of the set of output layers.

[0011] According to an embodiment, the determination includes determining that the first image should not be output based on a syntax element indicating an output layer set signaling mode, the mode including the following: the total number of output layer sets specified by the parameter set is equal to the maximum allowed number of layers in each encoded video sequence of the encoded video stream related to the parameter set, the i-th set of output layers from the sets of output layers includes layers with layer indices from 0 to i inclusive, and for each set of output layers from the number of sets of output layers shall be output only the highest layer in the set of output layers.

[0012] In an embodiment, the syntax element has a value of 0 to indicate the signaling mode of the output layer set.

[0013] According to an embodiment, the parameter set is a video parameter set.

[0014] According to an embodiment, the determination includes determining that the first image should not be output based on: a syntax element indicating the signaling mode of the set of output layers, a second image that belongs to the same access block as the first image, indicated as the subject output, the second image belongs to a higher layer than the first image, and the second image belongs to the output layer.

[0015] According to an embodiment, the determination includes determining that the first image should not be output based on a syntax element indicating an output layer set signaling mode, the mode including the following: the total number of output layer sets specified by the parameter set is explicitly signaled, the output layers of sets of output layers defined by the parameter set are explicitly signaled, and at least one of the output layers has a reference layer.

[0016] In an embodiment, the syntax element has a value of 2 to indicate the signaling mode of the output layer set.

[0017] According to an embodiment, the parameter set is a video parameter set.

[0018] According to an embodiment, the determination includes determining that the first image should not be output based on: a syntax element indicating the signaling mode of the output layer set, a parameter set identifier value greater than 0, and a second image that belongs to the same access block , the same as the first image, and designated as not belonging to any output layer.

[0019] According to an embodiment, the determination includes determining that the first image is to be output.

[0020] According to one or more embodiments, a system is provided for decoding an encoded video stream that includes video data divided into multiple layers. The system includes: a memory configured to store computer program code; and at least one processor configured to receive the encoded video stream, access the computer program code, and operate in accordance with instructions of the computer program code, the computer program code including: decoding code configured to cause the at least one processor to decode the video data encoded video stream, the decoding code includes a determination code configured to cause the at least one processor to determine whether a first image of the encoded video stream is to be output based on a syntax element that indicates a signaling mode of the set of output layers.

[0021] According to an embodiment, the determination code is configured to cause the at least one processor to determine that the first image should not be output based on a syntax element indicating an output layer set signaling mode, the mode including the following: total number of output layer sets , specified by a set of parameters is equal to the maximum allowable number of layers in each encoded video sequence of the encoded video stream related to the set of parameters, the i-th set of output layers from the sets of output layers includes layers with layer indices from 0 to i inclusive, and for each set of output layers Out of the output layer sets, only the highest layer in the output layer set should be output.

[0022] In an embodiment, the syntax element has a value of 0 to indicate the signaling mode of the output layer set.

[0023] In an embodiment, the determination code is configured to cause the at least one processor to determine that the first image should not be output based on: a syntax element indicating the signaling mode of the output layer set, a second image that belongs to the same access block as and the first image indicated as being output, the second image belongs to a higher layer than the first image, and the second image belongs to the output layer.

[0024] According to an embodiment, the determination code is configured to cause the at least one processor to determine that the first image should not be output based on a syntax element indicating an output layer set signaling mode, the mode including the following: total number of output layer sets , defined by the parameter set, is explicitly signaled, the output layers of the sets of output layers defined by the parameter set are explicitly signaled, and at least one of the output layers has a reference layer.

[0025] In an embodiment, the syntax element has a value of 2 to indicate the signaling mode of the output layer set.

[0026] According to an embodiment, the parameter set is a video parameter set.

[0027] In an embodiment, the determination code is configured to cause at least one processor to determine that the first image should not be output based on: a syntax element indicating the signaling mode of the output layer set, a parameter set identifier value greater than 0, and a second image , which belongs to the same access block as the first image, and is designated as not belonging to any output layer.

[0028] According to one or more embodiments, a non-transitory computer-readable medium is provided on which computer instructions are stored. The computer instructions are configured to cause at least one processor to: decode video data of an encoded video stream, wherein the video data is divided into a plurality of layers, wherein the decoding includes: determining whether a first image of the encoded video stream is to be output based on a syntax element that indicates an output set signaling mode layers.

BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Additional features, nature and various advantages of the disclosed invention will be better understood from the following detailed description and the accompanying drawings, in which:

[0030] In FIG. 1 is a schematic illustration of a simplified block diagram of a communication system in accordance with an embodiment.

[0031] In FIG. 2 is a schematic illustration of a simplified block diagram of a communication system in accordance with an embodiment.

[0032] In FIG. 3 is a schematic illustration of a simplified block diagram of a decoder in accordance with an embodiment.

[0033] In FIG. 4 is a schematic illustration of a simplified block diagram of an encoder in accordance with an embodiment.

[0034] In FIG. 5A is a schematic illustration of a first configuration for signaling ARC parameters in accordance with a prior art embodiment.

[0035] In FIG. 5B is a schematic illustration of a second configuration for signaling ARC parameters in accordance with a prior art embodiment.

[0036] In FIG. 6A is a schematic illustration of a first configuration for signaling ARC parameters in accordance with an embodiment.

[0037] In FIG. 6B is a schematic illustration of a second configuration for signaling ARC parameters in accordance with an embodiment.

[0038] In FIG. 6C is a schematic illustration of a third configuration for signaling ARC parameters in accordance with an embodiment.

[0039] In FIG. 7A is a schematic illustration of a tile group header excerpt according to an embodiment.

[0040] In FIG. 7B is a schematic illustration of an excerpt from a sequence parameter set in accordance with an embodiment.

[0041] In FIG. Figure 8 shows an example of a prediction structure for adaptive resolution scalability.

[0042] In FIG. 9A shows an example of a syntax table according to an embodiment.

[0043] In FIG. 9B illustrates an example of a syntax table according to an embodiment.

[0044] In FIG. 10 is a schematic illustration of a simplified block diagram of parsing and decoding a POC frame per access unit and access unit counter value in accordance with an embodiment.

[0045] In FIG. 11 is a schematic illustration of a video bitstream structure containing multi-layer sub-images in accordance with an embodiment.

[0046] In FIG. 12 is a schematic illustration of displaying a selected sub-image with improved resolution in accordance with an embodiment.

[0047] In FIG. 13 is a flowchart of a process for decoding and displaying a video bitstream containing multi-layer sub-images in accordance with an embodiment.

[0048] In FIG. 14 is a schematic illustration of a 360° video display with a sub-image enhancement layer in accordance with an embodiment.

[0049] In FIG. 15A shows an example of a layout of divided sub-images in accordance with an embodiment.

[0050] In FIG. 15B shows an example of a corresponding sub-image size and position information of one sub-image according to an embodiment.

[0051] In FIG. 16 shows a corresponding image prediction structure for the sub-images of FIG. 15A-B.

[0052] In FIG. 17 shows an example of an input image divided into a plurality of sub-regions, which may be encoded in one or more layers in accordance with an embodiment.

[0053] In FIG. 18 shows a corresponding layer and image prediction structure with a local area spatial scalability modality of the sub-regions illustrated in FIG. 17.

[0054] In FIG. 19A is a schematic illustration of an excerpt of a video parameter set in accordance with an embodiment.

[0055] In FIG. 19B is a schematic illustration of an excerpt from a sequence parameter set in accordance with an embodiment.

[0056] In FIG. 20 shows an example of a syntax table for sub-image layout information according to an embodiment.

[0057] In FIG. 21 shows an example of a syntax table for specifying output layers and profile/stage/level information for each set of output layers according to an embodiment.

[0058] In FIG. 22 shows an example of a syntax table to indicate an output layer mode for each set of output layers according to an embodiment.

[0059] In FIG. 23 shows an example syntax table for specifying the current sub-image of each layer for each set of output layers.

[0060] In FIG. 24 is a circuit diagram of a decoder according to an embodiment.

[0061] In FIG. 25 is a diagram of a computer system suitable for implementing the embodiments.

IMPLEMENTATION OF THE INVENTION

[0062] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present invention. The system (100) may include at least two terminals (110, 120) connected to each other via a network (150). For unidirectional data transmission, the first terminal (110) may encode video data at a local location for transmission to another terminal (120) via the network (150). The second terminal (120) may receive encoded video data of another terminal from the network (150), decode the encoded data, and display the reconstructed video data. Unidirectional data transfer may be common in media serving applications and the like.

[0063] FIG. 1 illustrates a second pair of terminals (130, 140) providing support for bidirectional encoded video transmission, which may occur, for example, during video conferencing. For bidirectional data transmission, each terminal (130, 140) may encode video data captured at a local location for transmission to another terminal via the network (150). Each terminal (130, 140) can also receive encoded video data transmitted by another terminal, can decode the encoded data, and can display the reconstructed video data on a local display device.

[0064] In FIG. 1, the terminals (110-140) may be illustrated as servers, personal computers and smartphones, and/or any other type of terminal. For example, the terminals (110-140) may be laptop computers, tablet computers, media players, and/or dedicated video conferencing equipment. Network (150) represents any number of networks that transmit encoded video data between terminals (110-140), including, for example, wired and/or wireless communication networks. The communications network (150) may communicate on circuit-switched and/or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For the purposes of this disclosure, the architecture and topology of the network (150) may not be essential to the operation of the present invention unless explained below herein.

[0065] FIG. 2 illustrates, as an example of an application for the disclosed invention, placement of a video encoder and decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, compressed video storage on digital media including CD, DVD, memory card, etc., and so on.

[0066] As shown in FIG. 2, the streaming system (200) may include a capture subsystem (213), which may include a video source (201) and an encoder (203). The video source (201) may be, for example, a digital camera, and may be configured to create an uncompressed stream (202) of video samples. The uncompressed video sample stream (202) may provide a large amount of data compared to encoded video bitstreams and may be processed by an encoder (203) connected to the camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (204) may include a smaller amount of data compared to the sample stream and may be stored on the streaming server (205) for future use. One or more streaming clients (206) may access the streaming server (205) to retrieve video bit streams (209), which may be copies of the encoded video bit stream (204).

[0067] In embodiments, the streaming server (205) may also function as a media information network element (MANE). For example, the streaming server (205) may be configured to reduce the encoded video bitstream (204) to adapt potentially different bitstreams to one or more streaming clients (206). In embodiments, the MANE may be provided separately from the streaming server (205) in the streaming system (200).

[0068] The streaming clients (206) may include a video decoder (210) and a display (212). The video decoder (210) may, for example, decode a video bit stream (209) that is an incoming copy of an encoded video bit stream (209) and create an output video sample stream (211) that can be displayed on a display (212) or other rendering device. (not shown). In some streaming systems, the video bit streams (204, 209) may be encoded according to certain video encoding/compression standards. Examples of such standards include, but are not limited to, ITU-T Recommendation H.265. A video encoding standard, informally known as Versatile Video Coding (VVC), is under development. Embodiments of the invention may be used in the context of VVC.

[0069] In FIG. 3 shows an exemplary functional block diagram of a video decoder (210) that is coupled to a display (212) according to an embodiment of the present invention.

[0070] The video decoder (210) may include a channel (312), a receiver (310), a buffer memory (315), an entropy decoder/analyzer (320), a scaler/deconverter (351), an intra- predictions, motion compensation prediction block (353), aggregator (355), loop filter block (356), reference image memory (357), and current image memory (). In at least one embodiment, the video decoder (210) may include an integrated circuit, a series of integrated circuits, and/or other electronic circuits. The video decoder (210) may also be partially or completely implemented as software running on one or more CPUs with associated memory.

[0071] In this embodiment and other embodiments, the receiver (310) may receive one or more encoded video sequences for decoding by the decoder (210) one encoded video sequence at a time, where the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequence may be received from a channel (312), which may be a hardware/software connection to a storage device in which the encoded video data is stored. The receiver (310) may receive the encoded video data along with other data, such as encoded audio data and/or ancillary data streams, which may be forwarded to their respective use objects (not shown). The receiver (310) can separate the encoded video sequence from other data. To combat network jitter, a buffer memory (315) may be coupled between the receiver (310) and the entropy decoder/analyzer (320) (hereinafter referred to as the “parser”). When the receiver (310) receives data from a storage/forwarding device with sufficient bandwidth and controllability, or from an iso-synchronous network, the buffer (315) may not be used or may be small. For use on best quality packet networks such as the Internet, a buffer (315) may be required and may be relatively large and may be of an adaptive size.

[0072] The video decoder (210) may include a parser (320) for recovering symbols (321) from the entropy encoded video sequence. Categories of these symbols include, for example, information used to control the operation of the decoder (210), and potentially information to control a visualization device, such as a display (212), that may be connected to the decoder, as shown in FIG. 2. The control information for the rendering device(s) may be in the form of, for example, additional enhancement information (SEI) messages or video usability information (VUI) parameter set tiles (not shown). The parser (320) may analyze/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be performed in accordance with a video encoding technology or standard and may follow principles well known to those skilled in the art, including variable length encoding, Huffman encoding, arithmetic encoding with or without context sensitivity, and so on. The parser (320) may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), and so on. The parser (320) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0073] The parser (320) may perform an entropy decoding/parsing operation on the video sequence received from the buffer (315) so as to create symbols (321).

[0074] The symbol reconstruction 321 may include several different blocks depending on the type of encoded video image or portions thereof (such as: inter- and intra-picture, inter- and intra-block) and other factors. Which blocks are engaged and how they are engaged can be controlled using subgroup control information that has been analyzed from the encoded video sequence by the analyzer (320). The flow of such subgroup control information between the analyzer (320) and the plurality of blocks is not shown below for clarity.

[0075] In addition to the functional blocks already mentioned, the decoder 210 can be conceptually divided into a number of functional blocks, as described below. In a practical implementation operating under commercial constraints, many of these devices interact closely with each other and can be at least partially integrated into each other. However, for purposes of describing the disclosed invention, it is appropriate to conceptually subdivide it into functional blocks as follows.

[0076] One block may be a scale/demap block (351). The scale/inverse transform block (351) may receive the quantized transform factor as well as control information including which transform to use, block size, quantization factor, quantization scaling matrices, etc. As symbol(s) (321) from the analyzer (320). A scale/invert block (351) may output blocks containing sample values that can be input to an aggregator (355).

[0077] In some cases, the output samples of the scaler/inverse transform (351) may refer to an intra-coded block; that is: a block that does not use prediction information from previously reconstructed images, but may use prediction information from previously reconstructed portions of the current image. Such prediction information may be provided by an intra-image prediction unit (352). In some cases, the image intra-prediction block (352) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information extracted from the current (partially reconstructed) image from the current image memory (358). The aggregator (355) in some cases adds, for each sample, prediction information generated by the intra-prediction block (352) to the output sample information provided by the upscaling/demapping block (351).

[0078] In other cases, the output samples of the scale/demap block (351) may be related to an intra-coded block and possibly a motion compensated block. In such a case, the motion compensated prediction unit (353) may access the reference image memory (357) to sample samples used for prediction. After motion compensation of the selected samples according to the symbols (321) associated with the block, these samples can be added by the aggregator (355) to the output of the scale/inverse block (351) (in this case called residual samples or residual signal) to generate information about output samples. The reference image memory addresses (357) from which the motion compensation predictor (353) selects prediction samples may be controlled by motion vectors. The motion vectors may be available to the motion compensation predictor 353 in the form of symbols 321, which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values retrieved from reference image memory (357) where precise sub-sample motion vectors, motion vector prediction mechanisms, and so on are used.

[0079] The output samples of the aggregator (355) may be subject to various loop filtering techniques in a loop filter block (356). Video compression technologies may include in-loop filtering technologies that are controlled by parameters included in the encoded video bitstream and provided to the loop filter block (356) as symbols (321) from the analyzer (320), but may also respond to meta-information received during the time of decoding previous (in decoding order) parts of the encoded image or encoded video sequence, as well as in response to previously reconstructed and loop-filtered sample values.

[0080] The output of the loop filter block (356) may be a stream of samples, which may be output to a rendering device such as a display (212) and also stored in reference image memory (357) for future use between inter-image predictions.

[0081] Certain encoded images, after complete restoration, can be used as reference images for future prediction. After the encoded image has been fully recovered and the encoded image has been identified as a reference image (eg, by the analyzer (320)), the current reference image may become part of the reference image memory (357), and the new current image memory may be reallocated before reconstruction begins. next encoded image.

[0082] The video decoder (210) may perform decoding operations according to a predefined video compression technology, which may be documented in a standard such as ITU-T Rec. H.265 An encoded video sequence may conform to the syntax specified by the video compression technology or standard being used, in the sense that it corresponds to the syntax of the video compression technology or standard as specified in the video compression technology document or standard and, in particular, in the profiles document therein . In addition, to comply with certain video compression technologies or standards, the complexity of the encoded video sequence may be within limits determined by the level of the video compression technology or standard. In some cases, levels limit the maximum image size, the maximum frame rate, the maximum restoration sampling rate (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the layers can in some cases be further limited by hypothetical reference decoder (HRD) specifications and HRD buffer management metadata carried in the encoded video sequence.

[0083] In an embodiment, the receiver (310) may receive additional (redundant) coded video data. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (210) to correctly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or SNR enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[0084] FIG. 4 illustrates an exemplary functional block diagram of a video encoder (203) associated with a video source (201), according to an embodiment of the present invention.

[0085] The video encoder (203) may include, for example, an encoder that is a source encoder (430), an encoding engine (432), a (local) decoder (433), a reference picture memory (434), a predictor (435), transmitter (440), entropy encoder (445), controller (450) and channel (460).

[0086] The encoder (203) may receive video samples from a video source (201) (which is not part of the encoder), which may capture video image(s) for encoding by the encoder (203).

[0087] The video source (201) may provide the original video sequence for encoding by the encoder (203) in the form of a discrete digital video signal stream, which may have any suitable bit depth (eg: 8 bits, 10 bits, 12 bits, ...), any color space (for example, VT.601 Y CrCB, RGB, ...) and any suitable sampling structure (for example, Y CrCb 4:2:0, Y CrCb 4:4:4). In a multimedia serving system, the video source (201) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (203) may be a camera that captures local image information as a video sequence. Video data may be provided as many individual images that, when viewed sequentially, convey motion. The images themselves can be organized as a spatial array of pixels, in which each pixel can contain one or more samples depending on the sampling structure, color space, etc. used. One skilled in the art can readily understand the relationship between pixels and samples. The description below focuses on the counts.

[0088] In an embodiment, the encoder (203) may encode and compress images of the original video sequence into an encoded video sequence (443) in real time or with any other timing constraints as required by the application. Providing an appropriate encoding rate is one of the functions of the controller (450). The controller (450) may also control other functional blocks, as described below, and may be operably coupled to these blocks. The connection is not shown for clarity. The parameters set by the controller (450) may include parameters related to speed control (image skip, quantizer, speed-distortion optimization methods lambda value, ...), image size, group of pictures (GOP) layout, maximum motion vector search range and so on. Other functions of the controller (450) can be readily recognized by one skilled in the art as they may relate to the video encoder (203) optimized for a particular system design.

[0089] Some video encoders operate in what one of ordinary skill in the art would readily recognize as an "encoding loop." As a simplified description, an encoding loop may consist of an encoding portion of a source encoder (430) (responsible for generating symbols based on the input image to be encoded and the reference image(s)), and a (local) decoder (433) built into an encoder (203) that reconstructs the symbols to create sample data that the (remote) decoder will also create when compression between the symbols and the encoded video bitstream is lossless in certain video compression technologies. This reconstructed sample stream may be entered into reference picture memory (434). Since decoding a symbol stream produces bit-accurate results regardless of the location of the decoder (local or remote), the contents of the reference picture memory are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values that the decoder "saw" when using the prediction during decoding. This fundamental principle of reference picture synchronicity (and the resulting drift if synchronicity cannot be maintained, for example due to channel errors) is known to one skilled in the art.

[0090] The operation of the "local" decoder (433) may be the same as that of the "remote" decoder (210), which was already described in detail above in the description of FIG. 3. However, since the symbols are available, and encoding/decoding of symbols into a coded video sequence by the entropy encoder (445) and analyzer (320) can be performed losslessly, the entropy decoding portion of the decoder (210), including the channel (312), receiver (310), the buffer (315) and parser (320) cannot be fully implemented in the local decoder (433).

[0091] An observation that can be made at this point is that any decoding technology other than parsing/entropy decoding that is present in a decoder may be required in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on the operation of the decoder. Descriptions of encoding technologies may be shortened because they may be the inverse of fully described decoding technologies. Only some areas require more detailed description, which is given below.

[0092] As part of its operation, the source encoder (430) may perform motion-compensated predictive encoding that predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence that have been designated “reference frames.” Thus, the encoding engine (432) encodes the differences between blocks of pixels of the input frame and blocks of pixels of the reference frame(s) that can be selected as the reference prediction(s) for the input frame.

[0093] The local video decoder (433) may decode encoded frame video data, which may be referred to as reference frames, based on the symbols generated by the source encoder (430). The operations of the encoding engine (432) may be predominantly lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can usually be a copy of the original video sequence with some errors. The local video decoder (433) copies the decoding processes that may be performed by the video decoder on the reference frames, and may cause the reconstructed reference frames to be stored in the reference picture memory (434). Thus, the encoder (203) can locally store copies of reconstructed reference frames that have common content as reconstructed reference frames that would be received by the far-end video decoder (assuming no transmission errors).

[0094] The predictor (435) may perform prediction searches for the encoding engine (432). That is, for a new frame to be encoded, the predictor (435) may search the reference picture memory (434) for sample data (as candidate pixel reference blocks) or certain metadata such as reference picture motion vectors, block shapes, etc. etc., and this can serve as a suitable support for predicting new images. The predictor (435) may operate on a block-by-pixel basis to find appropriate links to predict. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction links taken from a plurality of reference images stored in the reference image memory (434).

[0095] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[0096] The output of all the above functional blocks may be entropy encoded in an entropy encoder (445). The entropy encoder translates the symbols generated by the various functional blocks into an encoded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, and so on.

[0097] The transmitter (440) may buffer the encoded video sequences generated by the entropy encoder (445) to prepare them for transmission via a communications link (460), which may be a hardware/software link to a storage device that stores the encoded video data. The transmitter (440) may combine the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or ancillary data streams (sources not shown).

[0098] The controller (450) may control the operation of the encoder (203). During encoding, the controller (450) may assign each encoded image a specific encoded image type, which may affect the encoding techniques that can be applied to the corresponding image. For example, images can often be designated as an intra-image (Image), a prediction image (P-Image), or a bi-prediction image (B-Image).

[0099] An intra-picture (I-picture) may be one that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow various types of intra-pictures, including, for example, independent decoder refresh (IDR) pictures. The person skilled in the art is aware of these image variants I and their respective applications and features.

[0100] A prediction picture (P picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most one motion vector and a reference index to predict the sample values of each block.

[0101] A bidirectional prediction picture (B picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most two motion vectors and reference indices to predict sample values of each block. Similarly, multi-prediction images can use more than two reference images and associated metadata to reconstruct a single block.

[0102] Source images typically can be spatially divided into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. Blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the encoding assignment applied to the corresponding block images. For example, blocks of I pictures may be encoded without prediction, or they may be encoded predictively with reference to already encoded blocks of the same picture (spatial prediction or intra-prediction). The pixel blocks P of images may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one previously encoded reference picture. The picture blocks B may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one or two previously encoded reference pictures.

[0103] The video encoder (203) may perform encoding operations in accordance with a predetermined video encoding technology or standard, such as ITU-T Rec. H.265 In operation, the video encoder (203) may perform various compression operations, including predictive coding operations, that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore follow the syntax specified by the video encoding technology or standard being used.

[0104] In an embodiment, the transmitter (440) may transmit additional data with encoded video. The video encoder (430) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/signal-to-noise ratio enhancement layers, other forms of redundant data such as redundant images and slices, supplemental extended information (SEI) messages, visual usage information (VUI) tiles, and so on.

[0105] Before describing in more detail certain aspects of the embodiments of the invention, several terms are introduced below that are referred to in the remainder of this description.

[0106] "Sub-image" hereinafter refers in some cases to a rectangular arrangement of samples, blocks, macroblocks, encoding units, or similar objects that are semantically grouped and that can be independently encoded at a changed resolution. One or more sub-images may form an image. One or more coded sub-pictures may form a coded picture. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In certain environments, one or more encoded sub-images may be collected in a compressed domain without recoding to the sample level into the encoded image, and in the same or some other cases, one or more encoded sub-images may be extracted from the encoded image in the compressed domain.

[0107] “Adaptive resolution resampling” (ARC) hereinafter refers to mechanisms that allow the resolution of an image or sub-image to be changed in an encoded video sequence, for example, by resampling a reference image. “ARC parameters” hereinafter refer to control information necessary to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, output and/or reference image resolutions, various control flags, and so on.

[0108] The above description focuses on the encoding and decoding of a single semantically independent encoded video image. Before describing the consequences of encoding/decoding multiple sub-pictures with independent ARC parameters and the additional complexity implied, embodiments of ARC parameter signaling should be described.

[0109] In FIG. 6A-C show several new exemplary embodiments for signaling ARC parameters. As noted for each of the embodiments, they have certain advantages in terms of coding efficiency, complexity and architecture. A video coding standard or technology may implement one or more of these embodiments, and may also include embodiments known in the comparative art for signaling ARC parameters. Comparative embodiments include the examples illustrated in FIGS. 5A-B. The new embodiments may not be mutually exclusive and may possibly be included in a standard or technology that also includes prior art embodiments so that any of them can be used based on application needs, standards technology involved, or encoder choice.

[0110] ARC parameter classes may include: (1) upsampling/downsampling factors, individual or combined in the X and Y dimensions, or (2) upsampling/downsampling factors with the addition of a time dimension, indicating a constant rate of increase/decrease of a given amount images. Either of the above two may involve encoding or decoding one or more syntax elements that may indicate a table containing the factor(s). In embodiments, such syntax elements may be short in length.

[0111] "Resolution" may refer to the resolution in the X or Y dimension, in units of samples, blocks, macroblocks, CUs, or any other suitable granularity of the input image, output image, reference image, coded image, combined or separately. If there is more than one resolution (for example, one for the input image, one for the reference image), then in some cases one set of values can be inferred from another set of values. Permission can be limited, for example, using flags. A more detailed resolution example is provided below.

[0112] "Warp" coordinates similar to those used in Annex R H.263 may have a suitable granularity as described above. Annex R H.263 specifies one efficient way to encode such deformation coordinates, but other, potentially more efficient ways may also be used. For example, a reversible variable-length Huffman-style encoding for Annex P warp coordinates could be replaced by a suitable-length binary encoding, where the length of the binary codeword could, for example, be derived from the maximum image size, perhaps multiplied by a certain factor and offset by a specific value to allow for "warping" beyond the limits of the maximum image size.

[0113] With respect to upsampling or downsampling filter parameters, in the simplest case there may be only one filter for upsampling and/or downsampling. However, in some cases it may be preferable to provide greater flexibility in the filter design, which can be achieved by signaling the filter parameters. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g., by a list of filter coefficients using suitable entropy encoding techniques), and/or the filter may be selected implicitly by boost/buck ratios. samples that are signaled in accordance with any of the mechanisms mentioned above, and so on.

[0114] The following description assumes the exemplary case of encoding a finite set of upsampling/downsampling coefficients (the same coefficient must be used in both the X and Y dimensions), which are indicated by a codeword. This codeword may preferably be variable length encoded, for example, using Ext-Golomb code common to certain syntax elements in video encoding specifications such as H.264 and H.265. One suitable mapping of values to increase/decrease factors may, for example, correspond to Table 1 below.

[0115] Many similar mappings can be developed according to the needs of the application and the capabilities of the upscaling and downscaling mechanisms available in the video compression technology or standard. The table can be expanded to include more values. Values could also be represented by entropy encoding mechanisms other than Ext-Golomb codes (e.g. using binary encoding), which could have certain advantages where the resampling coefficients were of interest outside of the video processing engines themselves (primarily the encoder and decoder). for example, using MANE. It should be noted that for the (presumably) most common case where no resolution change is required, one can choose an Ext-Golomb code that is short (e.g. only one bit, such as shown in the second row of Table 1) which may have advantage in coding efficiency compared to using binary codes for the most common case.

[0116] The number of entries in the table, as well as their semantics, can be fully or partially customizable. For example, the underlying table structure may be conveyed in a "high" parameter set, such as a sequence or decoder parameter set. Alternatively or in addition, one or more such tables may be defined in a video coding technology or standard and may be selected, for example, by a decoder or sequence parameter set.

[0117] The following is a description of how the upsampling/downsampling factor (ARC information) encoded as described above can be included in a video encoding technology or standard syntax. Similar considerations may apply to one or more codewords driving the upsampling/downsampling filters. The following also describes when relatively large amounts of data may be required for a filter or other data structures.

[0118] As shown in FIG. 5A, Appendix P H.263 includes ARC information (502) in the form of four warp coordinates in the image header (501), specifically in the H.263 PLUSPTYPE header extension (503). This design may be reasonable when (a) there is an available image header and (b) frequent changes to the ARC information are expected. However, the overhead when using H.263 style signaling can be quite high, and the scaling factors may not apply to image boundaries since the image header may be temporary.

[0119] As shown in FIG. 5B, JVCET-M135-v1 includes an ARC reference information (505) (index) located in an image parameter set (504) that indexes a table (506) including target resolutions that is located within a sequence parameter set (507). Placing a possible resolution in a table (506) in a sequence parameter set (507) can be justified by using the SPS (507) as a point of interoperability negotiation during capability exchange. The resolution can be varied within the limits set by the values in the table (506) from image to image by accessing the appropriate set of image parameters (504).

[0120] As shown in FIG. 6A-C, the following embodiments of the present invention can transmit ARC information in a video bit stream, for example, to a decoder of the present invention. Each of these embodiments has certain advantages over the prior art described above. Embodiments may simultaneously be present in the same video coding technology or standard.

[0121] In the embodiment of FIG. 6A, ARC information (509), such as a resampling (scaling) factor, may be present in a header (508), such as, for example, a slice header, a GOB header, a tile header, or a tile group header. As an example, FIG. 6A illustrates a header (508) as a header for a group of tiles. This configuration may be adequate if the ARC information is small, such as a single variable-length codeword ue(v) or a fixed-length codeword of several bits, for example, as shown in Table 1. Having ARC information directly in the tile group header has the added benefit of the advantage is that the ARC information can be applied to a sub-image represented by, for example, a tile group corresponding to a tile group header, rather than to the entire image. Additionally, even if a video compression technology or standard only uses adaptive resolution changes of the entire image (as opposed to, for example, adaptive resolution changes based on tile groups), placing ARC information in the tile group header (for example, in H.263 - style picture header) has certain advantages in terms of error tolerance. Although the above description describes ARC information (509) present in a tile group header, it should be understood that the above description may similarly apply in cases where ARC information (509) is present, for example, in a slice header, GOB header or header mosaic element.

[0122] In the same or another embodiment, with reference to FIG. 6B, the ARC information (512) itself may be present in a corresponding parameter set (511), such as, for example, an image parameter set, a title parameter set, a tile parameter set, an adaptation parameter set, and so on. As an example, FIG. 6B illustrates a parameter set (511) as an adaptation parameter set (APS). The volume of this set of parameters may advantageously be no larger than the image. For example, the parameter set area could be a group of tiles. The use of ARC information (512) can be made implicit by activating the appropriate parameter set. For example, when a video coding technology or standard considers only image-based ARC, then an image parameter set or equivalent may be suitable as the corresponding parameter set.

[0123] In the same or different embodiment of FIG. 6C, ARC reference information (513) may be present in a tile group header (514) or similar data structure. The ARC reference information (513) may refer to a subset of the ARC information (515) available in the parameter set (516), with a scope beyond a single image. For example, the parameter set (516) may be a sequence parameter set (SPS) or a decoder parameter set (DPS).

[0124] The additional layer of indirection involves activating PPS from the tile group header, PPS or SPS, as used in JVET-M0135-v1, may be unnecessary since picture parameter sets, like sequence parameter sets, can be used to negotiate capabilities or ads. However, if the ARC information is to be applied to a sub-image that is also represented by, for example, a group(s) of tiles, a set of parameters (for example, a set of adaptation parameters or a set of header parameters) with an activation region limited to a group of tiles may be a better choice . Additionally, if the ARC information is more than negligible in size - for example, containing filter control information such as multiple filter coefficients - then the parameter may be a better choice than using the header directly in terms of encoding efficiency, since these settings can be re-used used by future images or sub-images referencing the same set of parameters.

[0125] When using a sequence parameter set or other higher level parameter set with a scope spanning multiple images, certain considerations may apply:

[0126] (1) A set (516) of parameters for storing ARC information (515) in a table may in some cases be a set of sequence parameters, but in other cases may be predominantly a set of decoder parameters. A decoder parameter set may have an activation region of several CVSs, namely the encoded video stream, that is, all encoded video bits from the start of the session to the end of the session. This scope may be more appropriate because possible ARC factors may be a decoder function, perhaps implemented in hardware, and hardware functions generally do not change with any CVS (which, at least in some entertainment systems, is a group images, one second or less in length). However, some embodiments may include an ARC information table in the sequence parameter set, as described herein, particularly in connection with point (2) below.

[0127] (2) The ARC reference information (513) may preferably be placed directly in the header (514) (e.g., image/slice/GOB/tile group header; hereinafter tile group header) rather than in the image parameter set, as in JVCET-M0135-v1, the reason is this: when the encoder wants to change one value in the image parameter set, such as for example the ARC reference information, then the encoder may have to create a new PPS and reference that new PPS. In the event that only the ARC reference information is changed but other information such as, for example, the quantization matrix information in the PPS remains, such information may be of significant size and will need to be retransmitted to make the new PPS complete. Since the ARC reference information may be a single codeword, such as an index in an ARC information table, which will be the only changing value, it would be cumbersome and wasteful to retransmit, for example, all of the quantization matrix information. Accordingly, placing the ARC reference information directly in the header (eg, header (514)) can be significantly better in terms of encoding efficiency, since indirect access through PPS, as proposed in JVET-M0135-v1, can be avoided. In addition, placing the ARC reference information in the PPS has the additional disadvantage that the ARC information referenced by the ARC reference information must necessarily apply to the entire image and not to a sub-image, since the activation volume of the image parameter set is the image.

[0128] In the same or another embodiment, ARC parameter signaling may follow a detailed example as shown in FIG. 7A-B. Fig. 7A-B in syntax diagrams. The notation of such syntax diagrams roughly corresponds to C-style programming. Lines in bold indicate syntax elements present in the bitstream, and lines without bold often indicate control flow or setting variables.

[0129] As an example of a header syntactic structure applicable to a (possibly rectangular) portion of an image, the tile group header (600) may conventionally comprise a variable-length Exp-Golomb encoded syntax element dec_pic_size_idx (602) (in bold). The presence of this syntax element in the tile group header (600) may be limited using adaptive resolution (603). Here, the value of the adaptive enable flag is not in bold, which means that the flag is present in the bitstream at the point where it occurs in the syntax diagram. Whether adaptive resolution is used for this image or parts thereof can be signaled in any high-level syntactic structure within or outside the bitstream. In the example of FIG. 7A-B, the adaptive resolution is signaled in the sequence parameter set (610), as shown below.

[0130] FIG. 7B illustrates an excerpt from a sequence parameter set (610). The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, such a flag may indicate the use of adaptive resolution, which in turn may require certain control information. In the example, such control information is present conditionally based on the value of a flag based on the if() statement (612) in the sequence parameter set (610) and the tile group header (600).

[0131] When adaptive resolution is used, in this example the output resolution (613) is encoded in sample units. Output resolution (613) in this exemplary embodiment refers to both syntax elements output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together may determine the resolution of the output image. Elsewhere in the video coding technology or standard, certain limits may be specified for any value. For example, the level definition may limit the number of total output samples that can be the product of the values of the above two syntax elements. In addition, certain video encoding technologies or standards, or external technologies or standards such as system standards, may limit the numbering range (for example, one or both dimensions must be divisible by a power of 2) or the aspect ratio (for example, width and height must be in a ratio such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementations or for other reasons.

[0132] In some applications, it may be desirable for the encoder to instruct the decoder to use a specific size of the reference image, rather than implicitly assuming the size to be the size of the output image. In this example, the reference_pic_size_present_flag syntax element (614) blocks the conditional presence of the reference picture dimensions (615) (again, the number refers to both the width and the height in the exemplary embodiment).

[0133] FIG. 7B further illustrates a table of possible decoding image widths and heights. Such a table may be expressed, for example, by a table reference (616) (eg, the syntax element num_dec_pic_size_in_luma_samples_minus1). The "minus 1" of a syntax element can refer to the interpretation of the meaning of that syntax element. For example, if the encoded value of a syntax element is zero, there is one entry in the table. If the coded value is five, there are six entries in the table. For each "row" in the table, the width and height of the decoded image are then included in the syntax as table entries (617).

[0134] The represented table entries (617) may be indexed using the dec_pic_size_idx syntax element (602) in the tile group header (600), thereby allowing different decoded sizes—essentially, scaling factors—for each tile group.

[0135] Certain video coding technologies or standards, such as VP9, support spatial scalability by implementing certain forms of reference picture re-sampling (which may be signaled completely differently than embodiments of the present invention) in combination with temporal scalability to provide spatial scalability. In particular, some reference images may be upsampled using ARC-style techniques to a higher resolution to form the basis of a spatial enhancement layer. Such upsampled images could be enhanced by conventional high-resolution prediction engines to add detail.

[0136] Embodiments of the invention can be used in such an environment. In some cases, in the same or another embodiment, a value in a network abstraction layer (NAL) block header, such as a temporal identifier field, may be used to indicate not only the temporal layer, but also the spatial layer. This provides certain benefits for certain systems; for example, existing Selected Forwarding Units (SFUs) created and optimized for forwarding selected at the temporal layer based on the value of the NAL block's temporary header identifier can be used without modification for scale-out environments. To make this possible, embodiments of the present disclosure may include a mapping between the encoded image size and the temporal layer, which must be indicated by a temporary identifier field in the NAL block header.

[0137] In some video encoding technologies, an access unit (AU) may refer to encoded images, slices, tiles, NAL units, etc. that have been captured and arranged into a corresponding image/slice/tile/NAL bitstream at a given moment in time. Such a point in time may be the time of composition.

[0138] In HEVC and some other video encoding technologies, a picture order counter (POC) value may be used to indicate a selected reference picture among a plurality of reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) contains one or more images, slices or tiles, each image, slice or tile belonging to the same AU may carry the same POC value from which it can be inferred that they were created from the contents of the same time composition. In other words, it can be determined that two images/slices/tiles belong to the same AU and have the same composition time in a scenario where two images/slices/tiles carry the same set POC value. Conversely, two images/slices/tiles having different POC values may indicate that these images/slices/tiles belong to different AUs and have different composition times.

[0139] In an embodiment of the invention, the above tight relationship may be relaxed since the access block may contain images, slices or tiles with different POC values. By allowing different POC values in the AU, it becomes possible to use the POC value to identify potentially independently decodable pictures/slices/tiles with identical presentation times. Accordingly, an embodiment of the present invention can support multiple scalable layers without changing reference picture selection signaling (eg, reference picture set signaling or reference picture list signaling), as described in more detail below.

[0140] In an embodiment, it is still desirable to be able to identify the AU to which an image/slice/tile belongs to another image/slice/tile having different POC values based only on the POC value. This can be achieved in the embodiments described below.

[0141] In the same or other embodiments, an access unit count (AUC) may be signaled in a high-level syntactic structure such as a NAL block header, a slice header, a tile group header, an SEI message, a parameter set, or an AU delimiter. The AUC value can be used to identify which NAL blocks, images, slices, or tiles belong to a given AU. The AUC value may correspond to a single point in time of the composition. The AUC value can be a multiple of the POC value. By dividing the POC value by an integer value, the AUC value can be calculated. In some cases, division operations may impose some overhead on decoder implementations. In such cases, slight restrictions in the numbering space of AUC values may allow the division operation to be replaced by the shift operations performed by embodiments of the present invention. For example, the AUC value may be equal to the value of the most significant bit (MSB) of the POC value range.

[0142] In the same embodiment, the POC cycle value on the AU (eg, PO_cycle_au syntax element) may be signaled in a high-level syntax structure such as a NAL block header, slice header, tile group header, SEI message, parameter set, or AU delimiter. The poc_cycle_au syntax elements can specify how many different and consecutive POC values can be associated with the same AU. For example, if the value of po_cycle_au is 4, images, slices, or tiles with a POC value of 0-3 inclusive are associated with an AU with an AUC value of 0, and images, slices, or tiles with a POC value between 4 and 7, inclusive, are associated with an AU with an AUC value of 1. Therefore, the AUC value can be derived by embodiments of the present invention by dividing the POC value by the POC cycle au value.

[0143] In the same or another embodiment, the poc_cycle_au value may be derived from information located, for example, in a video parameter set (VPS) that identifies the number of spatial layers or SNR layers in the encoded video sequence. This possible relationship is briefly described below. While output as described above may save a few bits in the VPS and therefore may improve encoding efficiency, it may be advantageous to explicitly encode poc_cycle_au in an appropriate high-level syntactic structure hierarchically below the video parameter set to be able to minimize poc_cycle_au for a given small portion of the bit stream, such as an image. This optimization can save more bits than can be stored by the inference process described above because POC values (and/or syntax element values indirectly related to POC) can be encoded in low-level syntax structures.

[0144] In the same or another embodiment of FIG. 9A illustrates an example syntax table to signal a syntax element vps_poc_cycle_au (632) in a VPS (630) or SPS that specifies the poc_cycle_au used for all pictures/slices in a coded video sequence, and FIG. 9B illustrates an example syntax table to signal a slice_poc_cycle_au syntax element (642) that indicates the poc_cycle_au of the current slice in the slice header (640). If the POC value increases uniformly for each AU, vps_contant_poc_cycle_per_au (634) in VPS (630) is set to 1, and vps_poc_cycle_au (632) is signaled in VPS (630). In this case, slice_poc_cycle_au (642) is not explicitly signaled and the AUC value for each AU is calculated by dividing the POC value by vps_poc_cycle_au (632). If the POC value does not increase uniformly per AU, vps_contant_poc_cycle_per_au (634) in VPS (630) is set to 0. In this case, vps_access_unit_cnt is not transmitted, while slice_access_unit_cnt is transmitted in the slice header for each slice or image. Each slice or image can have a different slice_access_unit_cnt value. The AUC value for each AU is calculated by dividing the POC value by slice_poc_cycle_au (642).

[0145] FIG. 10 illustrates a flowchart for describing a corresponding flowchart of the embodiment. For example, the decoder (or encoder) analyzes the VPS/SPS and determines whether the POC cycle on the AU is constant or not (652). The decoder (or encoder) then makes a decision 654 based on whether the POC cycle on the AU is constant in the encoded video sequence. That is, if the POC cycle on the AU is constant, the decoder (or encoder) calculates the access unit counter value from the sequence level poc_cycle_au value and the POC value (656). Alternatively, if the POC cycle on the AU is not constant, the decoder (or encoder) calculates the access unit counter value from the picture layer poc_cycle_au value and the POC value (658). In either case, the decoder (or encoder) can then repeat the process, for example by analyzing the VPS/SPS and determining whether the POC cycle on the AU is constant or not (662).

[0146] In the same or other embodiments, even though the POC value of an image, slice, or tile may be different, the image, slice, or tile corresponding to AUs with the same AUC value may be associated with the same decoding or instance withdrawal time. Therefore, without any mutual analysis/decoding dependency between pictures, slices or tiles in the same AU, all or a subset of the pictures, slices or tiles associated with the same AU can be decoded in parallel and can be output to the same time instance.

[0147] In the same or other embodiments, even though the POC value of an image, slice, or tile may be different, the image, slice, or tile corresponding to AUs with the same AUC value may be associated with the same decoding or instance time composition/display. When the composition time is contained in a container format, even if the images correspond to different AUs, as long as the images have the same composition time, the images can be displayed at the same point in time.

[0148] In the same or other embodiments, each image, slice, or tile may have the same temporal identifier (eg, temporal_id of the syntax element) in the same AU. All or a subset of the images, slices, or tiles corresponding to a point in time can be associated with the same time sublayer. In the same or other embodiments, each image, slice, or tile may have the same or a different spatial layer identifier (eg, layer_id of a syntax element) in the same AU. All or a subset of the images, slices, or tiles corresponding to a point in time may be associated with the same or a different spatial layer.

[0149] FIG. 8 illustrates an example of a video sequence structure (680) with a combination of temporal_id, layer_id, and adaptive resolution POC and AUC values. In this example, the image, slice, or tile in the first AU with AUC = 0 may have temporal_id = 0 and layer_id = 0 or 1, while the image, slice, or tile in the second AU with AUC = 1 may have temporal_id. = 1 and layer_id = 0 or 1 respectively. The POC value is increased by 1 for each image, regardless of the temporal_id and layer_id values. In this example, the value of poc_cycle_au may be equal to 2. In an embodiment, the value of poc_cycle_au may be set to the number of layers (spatial scalability). In this example, the POC value is increased by 2 and the AUC value is increased by 1. As an example, FIG. 8 illustrates, within the first AU (AUC = 0), an I slice (681) having POC 0, TID 0 and LID 0, and a B slice (682) having POC 1, TID 0 and LID. 1. In the second AU (AUC = 1) FIG. 8 illustrates a B-slice (683) having POC 2, TID 1 and LID 0, and a B-slice (684) having POC 3, TID 1 and LID 1. Within the third AU (AUC = 3) in FIG. 8 illustrates a B-slice (685) having POC 4, TID 0 and LID 0, and a B-slice (686) having POC 5, TID 0 and LID 1.

[0150] In the above embodiments, all or a subset of the inter-picture or inter-layer prediction structure and reference picture indication may be supported using existing reference picture set (RPS) signaling in HEVC or reference picture list (RPL) signaling. In RPS or RPL, the selected reference picture is indicated by signaling a POC value or a POC delta value between the current picture and the selected reference picture. In embodiments of the present invention, RPS and RPL can be used to indicate an inter-picture or inter-layer prediction structure without changing the signaling, but with the following limitations. If the temporal_id value of the reference picture is greater than the temporal_id value of the current picture, the current picture may not use the reference picture for motion compensation or other predictions. If the layer_id value of the reference image is greater than the layer_id value of the current image, the current image may not use the reference image for motion compensation or other predictions.

[0151] In the same and other embodiments, motion vector scaling based on the POC difference for temporal motion vector prediction may be disabled for a plurality of images in an access block. Therefore, although each image may have a different POC value in the access unit, the motion vector cannot be scaled and used for temporal prediction of the motion vector in the access unit, since a reference picture with a different POC in the same AU may be considered as a reference picture having the same moment of time. Therefore, in an embodiment, the motion vector scaling function may return 1 when the reference image belongs to an AU associated with the current image.

[0152] In the same and other embodiments, motion vector scaling based on the POC difference for temporal motion vector prediction may optionally be disabled for multiple images when the spatial resolution of the reference image differs from the spatial resolution of the current image. When motion vector scaling is enabled, the motion vector can be scaled based on both the POC difference and the spatial resolution ratio between the current image and the reference image.

[0153] In the same or another embodiment, the motion vector may be scaled based on the AUC difference instead of the POC difference to temporally predict the motion vector, especially when poc_cycle_au has a non-uniform value (when vps_contant_poc_cycle_per_au == 0). Otherwise (when vps_contant_poc_cycle_per_au == 1), scaling the motion vector based on the AUC difference may be identical to scaling the motion vector based on the POC difference.

[0154] In the same or another embodiment, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU (with the same AUC value) with the current image is not scaled based on the AUC difference and is used to predict the motion vector without scaling or scaling based on the spatial resolution ratio between the current image and the reference image.

[0155] In the same and other embodiments, the AUC value is used to identify the boundary of the AU and is used for a hypothetical reference decoder (HRD) operation that requires input and output synchronization to the AU granularity. In most cases, the decoded image with the highest layer in the AU can be output for display. The AUC value and layer_id value can be used to identify the output image.

[0156] In an embodiment, an image may comprise one or more sub-images. Each sub-image can cover a local area or the entire image area. The area supported by a sub-image may or may not overlap with the area supported by another sub-image. An area composed of one or more sub-images may or may not cover the entire image area. If an image consists of a sub-image, the region supported by the sub-image may be identical to the region supported by the image.

[0157] In the same embodiment, the sub-image may be encoded with an encoding method similar to the encoding method used for the encoded image. The sub-picture may be independently encoded or may be encoded depending on another standard picture or encoded picture. A sub-image may or may not have any analysis dependency on another standard image or a coded image.

[0158] In the same embodiment, the encoded sub-image may be contained in one or more layers. The encoded sub-image in a layer may have a different spatial resolution. The original sub-image may be spatially resampled (upsampled or downsampled), encoded at different spatial resolution parameters, and contained in a bitstream corresponding to the layer.

[0159] In the same or another embodiment, a sub-image with (W, H), where W denotes the width of the sub-image and H denotes the height of the sub-image, respectively, may be encoded and contained in the encoded bitstream corresponding to layer 0, while an upsampled (or downsampled) sub-image at the original spatial resolution, with (W*S _w , _k , H*S _{h, k} ), may be encoded and contained in the coded bitstream corresponding to layer k, where S _w,k , S _h,k indicate the horizontal and vertical resampling coefficients. If the values of S _{w, k} , S _{h, k} are greater than 1, resampling is equal to upsampling. Whereas, if the values of S _w,k , S _h,k are less than 1, resampling is equal to downsampling.

[0160] In the same or another embodiment, an encoded sub-image in a layer may have a different visual quality than an encoded sub-image in another layer in the same sub-image or another sub-image. For example, sub-image i in layer n is encoded by the quantization parameter Qi, _n , while sub-image j in layer m is encoded by the quantization parameter Oj, _m .

[0161] In the same or another embodiment, an encoded sub-image on a layer can be decoded independently, without any parsing or decoding dependence, from an encoded sub-image on another layer of the same local area. A sub-image layer that can be independently decoded without reference to another sub-image layer of the same local region is an independent sub-image layer. An encoded subimage on an independent subimage layer may or may not have any decoding or analysis dependence on a previously encoded subimage on the same subimage layer, but an encoded subimage may not have any dependence on an encoded image on another subimage layer.

[0162] In the same or another embodiment, an encoded sub-image on a layer can be decoded independently, without any parsing or decoding dependence, from an encoded sub-image on another layer of the same local area. A sub-image layer that can be independently decoded without reference to another sub-image layer of the same local region is an independent sub-image layer. An encoded subimage in a dependent subimage may reference an encoded subimage belonging to the same subimage, a previously encoded subimage in the same subimage layer, or both reference subimages.

[0163] In the same or another embodiment, the encoded sub-image contains one or more independent sub-image layers and one or more dependent sub-image layers. However, at least one independent sub-image layer may be present for the encoded sub-image. An independent sub-image layer may have a layer identifier value (eg, layer_id of a syntax element), which may be present in a NAL block header or other high-level syntax structure, equal to 0. A sub-image layer with a layer_id equal to 0 may be a base sub-image layer.

[0164] In the same or another embodiment, the image may include one or more foreground sub-images and one background sub-image. The area supported by the background subimage may be equal to the image area. The region supported by the foreground sub-image may overlap with the region supported by the background sub-image. The background sub-image may be a base layer of the sub-image, while the standard foreground image may be a non-base (enhancement) layer of the sub-image. One or more non-base layers of a sub-image may reference the same base layer for decoding. Each non-base subimage layer with layer_id equal to a can reference a non-base subimage layer with layer_id equal to b, where a is greater than b.

[0165] In the same or another embodiment, the image may contain one or more foreground sub-images with or without a background sub-image. Each sub-image can have its own base sub-image layer and one or more non-base (enhancement) layers. Each sub-image base layer may be referenced by one or more sub-image non-base layers. Each non-base subimage layer with layer_id equal to a can reference a non-base subimage layer with layer_id equal to b, where a is greater than b.

[0166] In the same or another embodiment, the image may contain one or more foreground sub-images with or without a background sub-image. Each coded sub-image in a (base or non-base) sub-image layer may be referenced by one or more non-base layer sub-images that belong to the same sub-image, and by one or more non-base layer sub-images that do not belong to the same sub-image.

[0167] In the same or another embodiment, the image may contain one or more foreground sub-images with or without a background sub-image. The sub-image in layer a can be further divided into multiple sub-images in the same layer. One or more encoded sub-images on layer b may reference the divided sub-images on layer a.

[0168] In the same or another embodiment, a coded video sequence (CVS) may be a group of coded pictures. A CVS may contain one or more coded sub-picture sequences (CSPS), where a CSPS may be a group of coded sub-pictures covering the same local image region. CSPS may have the same or a different temporal resolution than the encoded video sequence.

[0169] In the same or another embodiment, the CSPS may be encoded and contained in one or more layers. A CSPS may contain or be composed of one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS can reconstruct a sequence of sub-images corresponding to the same local region.

[0170] In the same or another embodiment, the number of CSPS layers corresponding to a CSPS may be identical to or different from the number of CSPS layers corresponding to another CSPS.

[0171] In the same or another embodiment, a CSPS layer may have a different temporal resolution (eg, frame rate) than another CSPS layer. The original (uncompressed) sub-image may be spatially resampled (upsampled or downsampled), encoded at different temporal resolution parameters, and contained in a bitstream corresponding to the layer.

[0172] In the same or another embodiment, a sub-image sequence at frame rate F may be encoded and contained in a coded bitstream corresponding to layer 0, while a time-upsampled (or downsampled) sub-image sequence from the original sub-image sequence wherein F*S _t,k may be encoded and contained in a coded bitstream corresponding to layer k, where S _t,k indicates a time sampling factor for layer k. If the value of S _{t, k} is greater than 1, the time sampling process is equal to frame rate upconversion. Whereas, if the value of S _{t, k} is less than 1, the temporal resampling process is equal to frame rate down-conversion.

[0173] In the same or another embodiment, when sub-images with CSPS layer a are referenced by a sub-image with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different based on the spatial resolution of layer bCSPS, the decoded pixels are In layer a, the CSPS are re-sampled and used as references. The resampling process may require filtering by upsampling or downsampling.

[0174] FIG. 11 illustrates an example video stream including a background CSPS video with a layer_id of 0 and several foreground CSPS layers. While the encoded sub-image may contain one or more CSPS enhancement layers (704), a background region that does not belong to any foreground CSPS layer may contain a base layer (702). The base layer (702) may contain a background region and foreground regions, while the CSPS enhancement layer (704) contains a foreground region. The CSPS enhancement layer (704) may have better visual quality than the base layer (702) in the same area. The CSPS enhancement layer (704) may reference the reconstructed pixels and motion vectors of the base layer (702) corresponding to the same region.

[0175] In the same or another embodiment, the video bit stream corresponding to the base layer (702) is contained in a track, while the CSPS layers (704) corresponding to each sub-picture are contained on a separate track in the video file.

[0176] In the same or another embodiment, the video bit stream corresponding to the base layer (702) is contained in a track, while the CSPS layers (704) with the same layer_id are contained in a separate track. In this example, the track corresponding to layer k includes only the CSPS layers (704) corresponding to layer k.

[0177] In the same or another embodiment, each CSPS layer (704) of each sub-image is stored on a separate track. Each track may or may not have parsing or decoding dependencies on one or more other tracks.

[0178] In the same or another embodiment, each track may contain bitstreams corresponding to layer i to layer j of the CSPS layers (704) of all or a subset of sub-images, where 0<i=<j=<k, where k is the highest layer CSPS.

[0179] In the same or another embodiment, the image contains or consists of one or more associated media data, including a depth map, an alpha map, 3D geometry data, an occupancy map, etc. Such associated synchronized media data may be divided into one or more sub-data streams, each corresponding to one sub-image.

[0180] In the same or another embodiment of FIG. 12 illustrates an example of a video conference based on a method for creating multi-layer sub-images. The video stream contains one base layer video bitstream corresponding to the background image and one or more enhancement layer video bitstreams corresponding to the foreground sub-images. Each enhancement layer video bitstream may correspond to a CSPS layer. The display shows the image corresponding to the base layer (712) by default. The base layer (712) may contain one or more user-in-picture (PIP) images. When a particular user is selected by the client control, the CSPS enhancement layer (714) corresponding to the selected user is decoded and displayed with improved quality or spatial resolution.

[0181] FIG. 13 illustrates an operating diagram of the embodiment. In an embodiment, the decoder may decode a video bitstream that includes multiple layers, such as, for example, one base layer and one or more CSPS enhancement layers (722). The decoder may then identify the background region and one or more foreground sub-images (724) and decide whether a particular sub-image region is selected (726). If a specific sub-image region corresponding to, for example, a user's PIP is selected (YES), the decoder may decode and display an enhanced sub-image corresponding to the selected user (728). For example, the decoder may decode and display an image corresponding to a CSPS enhancement layer (714). If no specific subimage region is selected (NO), the decoder may decode and display the background region (730). For example, the decoder may decode and display an image corresponding to the base layer (712).

[0182] In the same or another embodiment, a network intermediate unit (such as a router) may select a subset of layers to send to the user based on its bandwidth. The image/sub-image organization can be used to adapt the bandwidth. For example, if the user does not have the bandwidth, the router separates the layers or selects some sub-images based on their importance or based on the settings used. In an embodiment, such processes may be performed dynamically to adapt to bandwidth.

[0183] FIG. 14 illustrates an example of using 360-degree video. When a 360 degree spherical image (742) is projected onto a planar image, the 360 degree spherical image (742) that is projected can be divided into a plurality of sub-images (745) as a base layer (744). An enhancement layer (746) of a particular one of the sub-images (745) may be encoded and transmitted to the client. The decoder may decode both the base layer (744), including all sub-pictures (745), and the enhancement layer (746) of a selected one of the sub-pictures (745). When the current viewport is identical to the selected one of the sub-images (745), the displayed image may have a higher quality decoded sub-image (745) with an enhancement layer (746). Otherwise, the decoded image with the base layer (744) may be displayed with lower quality.

[0184] In the same or another embodiment, any layout information for display may be present in the file as additional information (eg, SEI message or metadata). One or more decoded sub-pictures may be moved and displayed depending on the transmitted arrangement information. The layout information may be provided by the streaming server or broadcaster, or may be reconstructed by a network entity or cloud server, or may be determined by individual user settings.

[0185] In an embodiment, when the input image is divided into one or more (rectangular) subregions, each subregion may be encoded as an independent layer. Each independent layer corresponding to a local area can have a unique layer_id value. For each independent layer, sub-image size and location information can be transmitted. For example, information about the image size (width, height) and the offset of the upper left corner (x_offset, y_offset) can be transmitted. Fig. 15A illustrates an example of the layout of divided sub-images (752), FIG. 15B illustrates an example of a corresponding sub-image size and position information of one of the sub-images (752), and FIG. 16 illustrates a corresponding image prediction structure. Layout information, including sub-image size(s) and sub-image position(s), may be signaled in a high-level syntactic structure such as a parameter set(s), a slice or tile group header, or an SEI message.

[0186] In the same embodiment, each sub-image corresponding to an independent layer may have its own unique POC value within the AU. When a reference picture from among the pictures stored in the DPB is specified using syntax element(s) in an RPS or RPL structure, the POC value(s) of each sub-picture corresponding to the layer may be used.

[0187] In the same or another embodiment, to indicate the (inter-layer) prediction structure, the layer_id may not be used and a POC (delta) value may be used.

[0188] In the same embodiment, a sub-image with a POC value of N corresponding to a layer (or local region) may or may not be used as a reference image of a sub-image with a POC value of K+N corresponding to the same layer (or that same local area) for motion compensated prediction. In most cases, the value of the number K can be equal to the maximum number of (independent) layers, which can be identical to the number of subregions.

[0189] In the same or another embodiment of FIG. 17-18 illustrate an expanded version of FIG. 15A-B and FIG. 16. When the input image is divided into several (for example, four) sub-regions, each local region can be encoded by one or more layers. In this case, the number of independent layers may be equal to the number of subregions, and one or more layers may correspond to a subregion. Thus, each subregion can be encoded by one or more independent layers and zero or more dependent layers.

[0190] In the same embodiment as shown in FIG. 17, the input image may be divided into four sub-regions, including a top-left sub-region (762), a top-right sub-region (763), a bottom-left sub-region (764) and a bottom-right sub-region (765). The right upper sub-region (763) can be encoded as two layers, which are layer 1 and layer 4, while the right lower sub-region (765) can be encoded as two layers, which are layer 3 and layer 5. In this case, the layer 4 may refer to layer 1 for motion compensated prediction, while layer 5 may refer to layer 3 for motion compensation.

[0191] In the same or another embodiment, in-loop filtering (eg, deblocking filtering, adaptive in-loop filtering, shapeshifting, two-way filtering, or any deep learning-based filtering) across a layer boundary can be (optionally) disabled.

[0192] In the same or another embodiment, motion compensated prediction or intra-block copying across a layer boundary may be (optionally) disabled.

[0193] In the same or another embodiment, boundary padding for motion-compensated prediction or in-loop filtering at the sub-image boundary may optionally be processed. A flag indicating whether boundary padding is processed or not may be signaled in a high-level syntactic structure, such as a parameter set(s) (VPS, SPS, PPS, or APS), a slice group or tile header, or an SEI message.

[0194] In the same or another embodiment, the layout information of the sub-area(s) (or sub-image(s)) may be transmitted to the VPS or SPS. Fig. 19A illustrates an example of syntax elements in the VPS (770), and FIG. 19B illustrates an example of SPS syntax elements (780). In this example, vps_sub_picture_dividing_flag (772) is passed to VPS (770). The flag may indicate whether the input images are divided into multiple subregions or not. When the value of vps_sub_picture_dividing_flag (772) is 0, the input picture(s) in the encoded video sequence(s) corresponding to the current VPS cannot be divided into multiple sub-pictures. In this case, the size of the input image may be equal to the size of the encoded image (pic_width_in_luma_samples (786), pic_height_in_luma_samples (788)), which is passed to the SPS (680). When the value of vps_sub_picture_dividing_flag (772) is 1, the input image(s) can be divided into multiple sub-pictures. In this case, the syntax elements vps_full_pic_width_in_luma_samples (774) and vps_full_pic_height_in_luma_samples (776) are passed to the VPS (770). The values of vps_full_pic_width_in_luma_samples (774) and vps_full_pic_height_in_luma_samples (776) can be equal to the width and height of the input image(s), respectively.

[0195] In the same embodiment, the values vps_full_pic_width_in_luma_samples (774) and vps_full_pic_height_in_luma_samples (776) cannot be used for decoding, but can be used for composition and display.

[0196] In the same embodiment, when the value of vps_sub_picture_dividing_flag (772) is 1, the syntax elements pic_offset_x (782) and pic_offset_y (784) can be passed to the SPS (780) that correspond to the specific layer(s). In this case, the size of the encoded image (pic_width_in_luma_samples (786), pic_height_in_luma_samples (788)) transmitted to the SPS (780) may be equal to the width and height of the sub-region corresponding to a particular layer. In addition, the position (pic_offset_x (782), pic_offset_y (784)) of the upper left corner of the sub-region may be signaled in the SPS (780).

[0197] In the same embodiment, the position information (pic_offset_x (782), pic_offset_y (784)) of the upper left corner of the sub-region cannot be used for decoding, but can be used for composition and display.

[0198] In the same or another embodiment, layout information (size and position) of all or a subset of subregion(s) of the input image(s), as well as dependency information between layers, may be conveyed to a parameter set or SEI message. Fig. 20 illustrates an example of syntax elements that indicate information about the layout of sub-regions, dependencies between layers, and the relationship between a sub-region and one or more layers. In this example, the num_sub_region (791) syntax element specifies the number of (rectangular) subregions in the current encoded video sequence. The num_layers syntax element (792) specifies the number of layers in the current encoded video sequence. The value of num_layers (792) can be equal to or greater than the value of num_sub_region (791). When any subregion is encoded as a single layer, the value of num_layers (792) may be equal to the value of num_sub_region (791). When one or more subregions are encoded as multiple layers, the value of num_layers (792) can be greater than the value of num_sub_region (791). The syntax element direct_dependency_flag [i] [j] (793) specifies the dependency from the j-th layer to the i-th layer. The syntax element num_layers_for_region[i] (794) specifies the number of layers associated with the i-th subregion. The syntax element sub_region_layer_id[i][j] (795) specifies the layer_id of the jth layer associated with the ith subregion. The syntax elements sub_region_offset_х [i] (796) and sub_region_offset_у [i] (797) indicate the horizontal and vertical position of the upper left corner of the i-th subregion, respectively. The syntax elements sub_region_width[i] (798) and sub_region_height[i] (799) specify the width and height of the i-th subregion, respectively.

[0199] In one embodiment, one or more syntax elements that define a set of output layers to indicate one or more layers to be output with or without profile row level information may be signaled in a high-level syntax structure (e.g., VPS, DPS , SPS, PPS, APS message or SEI). In fig. 21 syntax element num_output_layer_sets (804) indicating the number of output layer sets (OLS) in the encoded video sequence related to the VPS may be signaled to the VPS. For each set of output layers, the syntax element output_layer_flag (810) can be signaled as many times as there are output layers.

[0200] In the same embodiment, the output_layer_flag syntax element (810) equal to 1 indicates that the i-th layer is output. A syntax element output_layer_flag (810) equal to 0 indicates that the i-th layer is not output.

[0201] In the same or another embodiment, one or more syntax elements that define profile row level information for each set of output layers may be signaled in a high-level syntax structure (e.g., a VPS, DPS, SPS, PPS, APS, or SEI message ). In fig. 21 syntax elements num_profile_tier_level (806) indicating the amount of profile row information on the OLS in the encoded video sequence related to the VPS may be signaled to the VPS. For each set of output layers, a set of syntax elements for profile series level information or an index indicating specific profile series level information from among the entries in the profile series level information may be signaled as many times as there are output layers.

[0202] In the same embodiment, the syntax element profile_tier_level_idx[i][j] (812) specifies the index in the list of profile_tier_level() syntax structures (808) in the VPS of the profile_tier_level() syntax structure (808) that applies to the jth layer i-th MNC.

[0203] Profiles, rows and levels (and related information about them) may indicate restrictions on bitstreams and thus restrictions on the capabilities required to decode the bitstreams. Profiles, rows and levels (and associated information about them) can also be used to specify points of interaction between individual decoder implementations. A profile may be a subset of the entire bitstream syntax, such as a standard. Each profile (and associated information about it) can define a subset of algorithmic functions and constraints that can be supported by all decoders matching the profile. Levels and layers can be specified in each profile, and a row level can be a specified set of restrictions imposed on the values of syntax elements in the bitstream. Each row level (and associated information about it) may define a set of value limits and/or limits of arithmetic value combinations that can be accepted by decision syntactic elements. The same set of series and level definitions can be used with all profiles, but individual implementations may support a different series, and within a series, a different level for each supported profile. For any given profile, the row level can match the specific decoder processing load and memory capabilities. The level specified for a lower series may be more restrictive than the level specified for a higher series.

[0204] In the same or another embodiment, as shown in FIG. 22, syntax elements num_profile_tier_level (806) and/or num_output_layer_sets (804) may be signaled when the number of maximum layers is greater than 1 (vps_max_layers_minus1>0).

[0205] In the same or another embodiment, as shown in FIG. 22, a syntax element vps_output_layers_mode[i] (822) indicating the output layer signaling mode for the 1st set of output layers may be present in the VPS.

[0206] In the same embodiment, a syntax element vps_output_layers_mode [i] (822) equal to 0 indicates that only the highest layer is output with the i-th set of output layers. The vps_output_layers_mode [i] (822) syntax element equal to 1 indicates that all layers are output with the i-th set of output layers. A syntax element vps_output_layers_mode[i](822) equal to 2 indicates that the layers that are output are those with vps_output_layer_flag[i][j] equal to 1 with the i-th output layer set. Other values may be reserved.

[0207] In the same embodiment, the output_layer_flag [i] [j] syntax element (810) may or may not be signaled depending on the value of the vps_output_layers_mode [i] syntax element (822) for the i-th set of output layers.

[0208] In the same or another embodiment, as shown in FIG. 22, the flag vps_ptl_signal_flag[i] (824) may be present for the i-th set of output layers. Depending on the value of vps_ptl_signal_flag [i] (824), the profile row level information for the i-th set of output layers may or may not be transmitted.

[0209] In the same or another embodiment, as shown in FIG. 23, the number of subpics, max_subpics_minus1, in the current CVS may be signaled in a high-level syntax structure (eg, in a VPS, DPS, SPS, PPS, APS, or SEI message).

[0210] In the same embodiment as shown in FIG. 23, subpicture ID, sub_pic_id[i] (821), for the i-th subpicture can be signaled when the number of subpictures is greater than 1 (max_subpics_minus1>0).

[0211] In the same or another embodiment, one or more syntax elements indicating a sub-image identifier belonging to each layer of each set of output layers may be signaled to the VPS. In fig. 23 identifier sub_pic_id_layer [i] [j] [k] (826) indicates the k-e sub-image present in the j-th layer of the i-th set of output layers. Using the identifier information sub_pic_id_layer[i][j][k] (826), the decoder can recognize which sub-picture can be decoded and output for each layer of a particular set of output layers.

[0212] In an embodiment, the TargetOlsIdx variable, which identifies the OLS index of the target OLS to be decoded, and the Htid variable, which identifies the highest temporal sublayer to be decoded, are set by some external means. The BitstreamToDecode bitstream may not contain any other layers than those included in the target OLS, and may not include any NAL block with a Temporalid greater than Htid.

[0213] In the same or another embodiment, ols_output_layer_flag [i] [j] equal to 1 indicates that the layer with nuh_layer_id equal to vps_layer_id [j] is the output layer of the i-th OLS when ols_mode_idc (864) is equal to 2. The ols_output_layer_flag[i][j] syntax element equal to 0 indicates that the layer with nuh_layer_id equal to vps_layer_id[j] is not the output layer of the i-th OLS when the ols_mode_idc syntax element is equal to 2. The nuh_layer_id syntax element can specify a layer identifier, to which the non-video coding layer (VCL) NAL block belongs, or the identifier of the layer to which the non-VCL NAL block applies. The value of nuh_layer_id may be the same for all VCL NAL blocks of the encoded image. The nuh_layer_id value of a coded picture or picture unit (PU) may be the nuh_layer_id value of the VCL NAL blocks of the coded picture or PU. The elemetn nal_unit_type syntax value may be the same for all coded video sequence (CVSS) start pictures of the AU.

[0214] NumOutputLayersInOls [i] variable indicating the number of output layers in the i-th OLS, NumSubLayersInLayerlnOLS [i] [j] variable indicating the number of sublayers in the j-th layer in the i-th OLS, OutputLayerldlnOls [i] [j] variable by defining the nuh_layer_id value of the j-th output layer in the i-th OLS, and the variable LayerUsedAsOutputLayerFlag[k] determining whether the k-th layer is used as an output layer in at least one OLS, can be obtained as follows:

[0215] For each value of i in the range 0 to vps_max_layers_minus 1 inclusive, the values of LayerUsedAsRefLayerFlag [i] and LayerUsedAsOutputLayerFlag [i] cannot be equal to 0 at the same time. In other words, there may be no layer that is neither the output layer of at least one OLS nor the direct reference layer of any other layer.

[0216] For each OLS there may be at least one layer that is an output layer. In other words, for any value of i in the range 0 to TotalNumOlss - 1 inclusive, the value of NumOutputLayersInOls[i] can be greater than or equal to 1.

[0217] It is possible to obtain the variable NumLayersInOls [i] defining the number of layers in the i-th OLS, and the variable LayerldlnOls [i] [j] defining the nuh_layer_id value of the j-th layer in the i-th OLS, as follows:

[0218]

The 0th OLS can only contain the bottommost layer (i.e. the layer with nuh_layer_id equal to vps_layer_id[0]), and for the 0th OLS only the included layer can be output.

[0219] The OlsLayerldx[i][j] variable defining the OLS layer index for a layer with nuh_layer_id equal to LayerldlnOls[i][j] is obtained as follows:

[0220] The lowest layer in each OLS may be an independent layer. In other words, for every i in the range 0 to TotalNumOlss - 1 inclusive, the value of vps_independent_layer_flag [GeneralLayerldx [LayerldlnOls [i] [0]]] can be equal to 1.

[0221] Each layer may be included in at least one OLS defined by the VPS. In other words, for each layer with a particular nuh_layer_id value, nuhLayerId, equal to one of vps_layer_id [k] for k in the range 0 to vps_max_layers_minus1 inclusive, there can be at least one pair of values i and j, where i is in the range 0 to TotalNumOlss - 1 inclusive, and j is in the range NumLayersInOls[i] - 1 inclusive, so the value of LayerldInOls[i][j] is equal to nuhLayerId.

[0222] An ols_mode_idc syntax element (864) equal to 0 may indicate that the total number of OLSs defined by the VPS is vps_max_layers_minusl+1, the i-th OLS includes layers with layer indices from 0 to i inclusive, and for each OLS is output only at the highest OLS layer.

[0223] An ols_mode_idc syntax element (864) equal to 1 may indicate that the total number of OLSs defined by the VPS is vps_max_layers_minus 1+1, the i-th OLS includes layers with layer indices from 0 to i inclusive, and for each OLS displays all OLS layers.

[0224] An ols_mode_idc syntax element (864) equal to 2 may indicate that the total number of OLSs specified by the VPS are explicitly signaled, and for each OLS, the output layers are explicitly signaled, and the other layers are layers that are direct or indirect output reference layers OLS layers.

[0225] The value of the ols_mode_idc syntax element (864) can range from 0 to 2, inclusive. Syntax element value 3 ols_mode_idc (864) may be reserved for future use by ITU-T|ISO/IEC.

[0226] In the same or another embodiment, decoding processes may be performed as described below.

[0227] Bross, et. al, “Universal Video Coding (Draft 8),” Joint Video Experts Group paper JVET-Q2001-vA, January 7-17, 2020, is incorporated herein in its entirety and is hereafter referred to as “Paper 1.”

[0228] Decoding processes may be applied to each encoded image in the BitstreamToDecode bitstream. Each encoded picture may be called the current picture and denoted by the variable CurrPic. The bitstream BitstreamToDecode can be an input to decoding processes.

[0229] The chroma_format_idc syntax element may be obtained by an encoder or a decoder. The chroma_format_idc syntax element can specify chroma sampling relative to luma sampling. Depending on the value of the chromaformatidc syntax element, the number of sample arrays of the current image can be (for example, determined by the decoder) as follows:

[0230] (A) If the chroma_format_idc syntax element is 0, the current image consists of 1 sample array _SL .

[0231] (B) Otherwise (chroma_format_idc is not equal to 0) the current image consists of 3 arrays of samples SL, _SCb,SCr .

[0232] The current picture decoding process may take as input the syntax elements and uppercase variables from Document 1 Section 7. In interpreting the semantics of each syntax element in each NAL block and the rest of Document 1 Section 8, the term "bitstream" (or part of it, e.g. CVS bitstream) refers to BitstreamToDecode (or part thereof).

[0233] The separate color_jt_lane_flag syntax element may indicate whether the image is encoded using separate color planes. For example, when the syntax element element selected_colour_plane_flag is 1, the syntax element can indicate that the image is encoded using three separate color planes. When an image is encoded using three separate color planes, a slice may contain only coding tree units (CTUs) of a single color component, identified by a corresponding color_plane_id value, and each array of image color components may consist of slices having the same color_plane_id value. Coded slices with different color_jt_lane_id values in an image may be interleaved with each other with the limitation that for each color_plane_id value, coded slice NALs with that colou_plane_id value may be in ascending order of CTU address in the order of tile scans for the first CTU of each coded slice NAL. When the separate_colour_plane_flag syntax element is 1, each color component CTU can be contained in exactly one slice (i.e., the information for each image CTU is present in exactly three slices, and those three slices have different color_plane_id values). When the separate color_plane_flag syntax element is 0, each CTU of an image can be contained in exactly one slice.

[0234] Depending on the value of the separate_colour_plane_flag syntax element, the decoding process may be structured as follows:

[0235] (A) If separate_colour_plane_flag is 0, the decoding process is started once with the current image being the output.

[0236] (B) Otherwise (separate_colour_plane_flag is 1), the decoding process can be started three times. The inputs to the decoding process are all NAL blocks of the encoded image with an identical color_plane_id value. The process of decoding NAL blocks with a particular color_plane_id value can be defined as if only a monochrome color format CVS with that particular color_plane_id value was present in the bitstream. The output of each of the three decoding processes can be assigned to one of three current image sample arrays, with NAL blocks with color_plane_id equal to 0, 1 and 2 being assigned to S _L , S _Cb and S _Cr respectively.

[0237] The ChromaArrayType variable may be returned as 0 if the separate_colour_plane_flag syntax element is 1 and the chroma_format_idc is 3. During decoding, the value of the ChromaArrayType variable may be evaluated, which may result in operations identical to those for monochrome images (when chroma_format_idc is 0 ).

[0238] For the current CurrPic picture, the decoding process may be performed as follows:

[0239] (A) Decoding of NAL blocks may be performed in accordance with clause 8.2 of Document 1. For example, decoding inputs may be NAL blocks of current pictures and associated non-VCL NAL blocks. The decoding output can be parsed by byte sequence payload (RBSP) syntactic structures encapsulated in NAL blocks. Decoding for each NAL block can extract the RBSP syntax structure from the NAL block, and then parse the RBSP syntax structure.

[0240] (B) Decoding processes using syntax elements at the slice header layer and above may be performed in accordance with, for example, clause 8.3 of Document 1:

[0241] (1) Variables and functions related to image order calculation can be obtained in accordance with section 8.3.1 of Document 1. This can only be called for the first image slice.

[0242] (2) At the beginning of the decoding process, for each non-IDR image slice, a decoding process may be started to construct reference picture lists to obtain reference picture list 0 (RefPicList [0]) and reference picture list 1 (RefPicList [1]) in in accordance with paragraph 8.3.2 of Document 1.

[0243] (3) The decoding process for marking reference pictures may be performed in accordance with clause 8.3.3 of Document 1, in which reference pictures may be marked as “not used for reference” or “used for long-term reference”. The decoding process for marking a reference image can only be started for the first image slice.

[0244] (4) When the current picture is a CRA picture with NoOutputBeforeRecoveryFlag equal to 1 or a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, a decoding process may be started to generate unavailable reference pictures in accordance with subclause 8.3.4 of Document 1, which may be called only for the first image slice.

[0245] (5) PictureOutputFlag can be set (eg by a decoder) as follows:

[0246] If one of the following conditions is determined to be true, the PictureOutputFlag of the current image can be set to 0 so that the image is marked as "not required for output" and is not output:

[0247] (a) The current image is a RASL image and the NoOutputBeforeRecoveryFlag syntax element of the associated IRAP image is 1.

[0248] (b) The syntax element gdr_enabled_flag is 1, and the current image is a GDR image with NoOutputBeforeRecoveryFlag equal to 1.

[0249] (c) The syntax element gdr_enabled_flag is 1, the current image is associated with a GDR image with NoOutputBeforeRecoveryFlag equal to 1, and the PicOrderCntVal of the current image is less than the RpPicOrderCntVal of the associated GDR image.

[0250] (d) The sps_video_parameter_set_id syntax element is greater than 0, ols_mode_idc is equal to 0, and the current AU contains a picture picA that satisfies all of the following conditions: PicA has a PictureOutputFlag equal to 1; PicA has a nuh_layer_id nuhLid greater than that of the current image; and PicA belongs to the output layer (for example, ols_output_layer_flag [TargetOlsIdx] [GeneralLayerldx [nuh_layer_id]] is 1).

[0251] (e) The sps_video_parameter_set_id syntax element is greater than 0, ols_mode_idc is equal to 2, and the ols_output_layer_flag [TargetOlsIdx] [GeneralLayerldx [nuh_layer_id]] syntax element is 0.

[0252] Otherwise, PictureOutputFlag is set to pic_output_flag.

[0253] Decoding processes may be performed using syntax elements on all layers of the syntax structure in accordance with clauses 8.4, 8.5, 8.6, 8.7 and 8.8 of Document 1. For example, a decoding process may be performed for encoding blocks encoded in intra-prediction mode, mode inter-prediction and IBC prediction mode; array scaling, transformation, and processing processes can be performed; and in-loop filtration processes can be performed. A bitstream match may be required so that the encoded image slices contain slice data for each CTU of the image, such that dividing the image into slices and dividing the slices into CTUs constitutes a division of each image.

[0254] After all slices of the current image have been decoded, the current decoded image may be marked as "used for short-term reference", and each ILRP entry in RefPicList[0] or RefPicList[1] can be marked as "used for short-term reference". certificates."

[0255] The GeneralLayerldx [i] variable defining the layer index for the layer with nuh_layer_id equal to vps_layer_id [i] can be obtained as follows:

[0256] According to one or more embodiments, NAL blocks and syntax elements therein (eg, those described herein) may be received by decoders of the present invention to decode video data from an encoded video stream. Decoders of the present invention can decode video data from an encoded video stream. For example, as shown in FIG. 24, the decoder (880) may include decoding code (885) configured to cause at least one decoder processor (880) to decode the video data. According to embodiments, the decoding code (885) may comprise a determination code (887) configured to cause the at least one processor to determine whether a first image of the encoded video stream should be output (eg, PictureOutputFlag=0 or 1) based on the syntax element ( for example, ols_mode_idc), which indicates the mode (for example, ols_mode_idc=0 or ols_mode_idc=2) of the output layer set signaling. Alternatively or additionally, the determination code (885) may be configured to cause the at least one processor to determine whether a first image of the encoded video stream should be output based on other conditions described in the present invention.

[0257] The methods described above may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 25 illustrates a computer system (900) suitable for implementing embodiments of the disclosed invention.

[0258] Computer software may be encoded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code containing instructions that may be executed directly or through interpretation, execution of microcode, etc. .P. computer central processing units (CPU), graphics processing units (GPU), etc.

[0259] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0260] The components shown in FIG. 25 for a computer system (900) are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of computer software implementing embodiments of the present invention. The component configuration should also not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary computer system embodiment (900).

[0261] The computer system (900) may include certain human interface input devices. Such a human interface input device may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movement), audio input (e.g., voice, hand clapping), visual input, (eg: gestures), olfactory input (not shown). Human interface devices can also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, ambient sound), images (e.g. scanned images, photographic images obtained from a still camera images), video (e.g. 2D video, 3D video, including stereoscopic video).

[0262] Human input devices may include one or more of (only one of each illustrated): keyboard (901), mouse (902), trackpad (903), touch screen (910), data glove, joystick (905), microphone (906), scanner (907) and camera (908).

[0263] The computer system (900) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (910), data glove, or joystick (905), but there may also be haptic feedback devices that do not serve as input devices). For example, such devices may be audio output devices (such as speakers (909), headphones (not shown)), image output devices (for example, screens (910), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without haptic feedback capability, some of which can provide two-dimensional visual output or more three-dimensional output through means such as stereographic output; virtual reality glasses (not shown ), holographic displays and smoke tanks (not pictured)), and printers (not pictured).

[0264] The computer system (900) may also include human accessible storage devices and associated media, such as optical media, including CD/DVD ROM/RW (920) from CD/DVD or similar media (921), flash drive (922), removable hard drive or solid state drive (923), legacy magnetic media such as tape and floppy (not shown), ROM/ASIC/PLD-based specialized devices such as dongles (not shown), and etc.

[0265] Those skilled in the art will also understand that the term “computer readable medium” as used in connection with the subject matter disclosed herein does not cover transmission media, carrier waves, or other time signals.

[0266] The computer system (900) may also include an interface to one or more communications networks. Networks can be, for example, wireless, wired, optical. In addition, networks can be local, global, metropolitan, automotive and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired television or wireless wide area digital networks including cable television, satellite television and terrestrial broadcasting. Television, automotive and industrial equipment, including CANBus, etc. Some networks typically require external network interface adapters that are connected to certain general purpose data ports or peripheral buses (949) (such as the USB ports of a computer system (900); others are typically integrated into the core of a computer system 900) by connecting to the system bus. as described below (for example, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (900) can communicate with other entities. Such communications may be unidirectional, receive-only (such as broadcast television), unidirectional, send-only (such as CANbus to certain CANbus devices), or bidirectional, such as for other computer systems using local or wide area digital networks. Such communication may include exchanging data with a cloud computing environment (955). Specific protocols and protocol stacks may be used on each of these networks and network interfaces, as described above.

[0267] The aforementioned human interface devices, human accessible storage devices, and network interfaces (954) may be attached to the core (940) of the computer system (900).

[0268] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), field programmable gate area (FPGA) processors (943), hardware, accelerators (944) for specific tasks, etc. These devices, along with read-only memory (ROM) (945), random access memory (946), internal storage devices such as internal hard drives not accessible to the user, solid-state drives, etc. (947) can be connected via the system bus (948). On some computer systems, the system bus (948) may be available in the form of one or more physical connectors to allow expansion with additional processors, graphics processors, and the like. Peripherals can be connected either directly to the core system bus (948) or via a peripheral bus (949). Peripheral bus architecture includes PCI, USB, etc. A 950 graphics adapter can be included in a 940 core.

[0269] The CPU (941), GPU (942), FPGA (943), and accelerators (944) may execute certain instructions that, when combined, may constitute the aforementioned computer code. This computer code may be stored in ROM (945) or RAM (946). Transient data may also be stored in RAM (946), while persistent data may be stored, for example, in internal storage (947). Fast storage and retrieval to any of the storage devices can be enabled through the use of cache memory, which can be tightly coupled to one or more CPU (941), GPU (942), mass storage (947), ROM (945), RAM (946) and the like.

[0270] The computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specifically designed and constructed for the purposes of the present invention, or they may be well known and available to those skilled in the computer software art.

[0271] By way of example, and not limitation, a computer system having an architecture (900) and, in particular, a core (940) may provide functionality through processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.). etc.) executing software embodied on one or more tangible machine-readable media. Such computer-readable media may be media associated with user-accessible mass storage, as described above, as well as certain kernel storage (940) that is non-volatile in nature, such as internal mass storage (947) or ROM (945). Software implementing various embodiments of the present invention may be stored in such devices and executed by the kernel (940). The computer-readable medium may include one or more storage devices or chips according to specific needs. Software may cause the core (940) and, in particular, the processors therein (including CPUs, GPUs, FGPAs, etc.) to perform certain processes or certain parts of specific processes described herein, including defining data structures stored in RAM (946), and modification of such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic built into hardware circuitry or otherwise embodied in circuitry (e.g., accelerator (944)) that may operate in place of or in conjunction with software to perform specific processes or portions of specific processes. described in this document. A reference to software may include logic and vice versa where appropriate. Reference to a computer-readable medium may cover circuitry (eg, an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both, where appropriate. The present invention embraces any suitable combination of hardware and software.

[0272] Although several non-limiting exemplary embodiments are described herein, there are variations, permutations, and various substitute equivalents that fall within the scope of the present invention. Thus, it will be appreciated that those skilled in the art will be able to develop numerous systems and methods that, although not expressly shown or described herein, embody the principles of the invention and are thus within the spirit and scope thereof.

Claims (15)

1. A method for decoding an encoded video stream, performed by at least one processor, comprising the following steps: receiving an encoded video stream including video data divided into a plurality of layers; and decoding video data of the encoded video stream, wherein the decoding comprises: determining whether a first image of the encoded video stream is to be output based on a single syntax element indicating a signaling mode of the set of output layers, wherein the determination comprises determining that the first image is not to be output based on the single element syntax indicating the signaling mode of the output layer set, and the mode includes the following: the total number of output layer sets defined by the parameter set is equal to the maximum allowed number of layers in each coded video sequence of the coded video stream associated with the parameter set, wherein the i-th output layer set among the output layer sets includes layers with layer indices from 0 to i, inclusive, and for each output layer set, only the highest layer in the output layer set, or the total number of output layer sets defined by the parameter set, shall be output from among the output layer sets , is explicitly signaled, the output layers of the sets of output layers defined by the parameter set are explicitly signaled, and at least one of the output layers has a reference layer. 2. The method of claim 1, wherein the single syntax element has a value of 0 to indicate the signaling mode of the set of output layers. 3. The method according to claim 1, wherein the set of parameters is a set of video parameters. 4. The method of claim 1, wherein the determination includes determining that the first image should not be output based on: a single syntax element indicating the signaling mode of the set of output layers, a second image belonging to the same access unit as the first image, indicated as to be output, a second image belonging to a higher layer than the first image, and a second image belonging to the output layer. 5. The method of claim 1, wherein the single syntax element has a value of 2 to indicate the signaling mode of the set of output layers. 6. The method according to claim 1, wherein the set of parameters is a set of video parameters. 7. The method of claim 1, wherein the definition comprises a determination that the first image shall not be output based on: a single syntax element indicating the signaling mode of the output layer set, a parameter set identifier value greater than 0, and a second image belonging to the same access block , as the first image, indicated as not belonging to any output layer. 8. A system for decoding an encoded video stream including video data divided into a plurality of layers, the system comprising: a memory configured to store computer program code; and at least one processor configured to receive the encoded video stream, access the computer program code, and operate in accordance with the instructions of the computer program code, the computer program code comprising: decoding code configured to cause the at least one processor to decode video data of the encoded video stream, wherein the decoding code comprises a determination code configured to cause the at least one processor to determine whether a first image of the encoded video stream should be output based on a single syntax element that indicates a signaling mode of the set of output layers, wherein the determination code is configured to cause at least one processor to determine that the first image should not be output based on a single syntax element indicating the signaling mode of the output layer set, and the mode includes the following: the total number of output layer sets defined by the parameter set is equal to the maximum allowed number of layers in each encoded video sequence an encoded video stream related to a set of parameters, wherein the i-th set of output layers from among the sets of output layers includes layers with layer indices from 0 to i inclusive, and for each set of output layers from among the sets of output layers only the highest layer is output in the set of output layers, or the total number of sets of output layers defined by the parameter set, is explicitly signaled, the output layers of the sets of output layers defined by the parameter set are explicitly signaled, and at least one of the output layers has a reference layer. 9. The system of claim 8, wherein the single syntax element has a value of 0 to indicate the signaling mode of the set of output layers. 10. The system of claim 8, wherein the set of parameters is a set of video parameters. 11. The system of claim 8, wherein the determination code is configured to cause the at least one processor to determine that the first image should not be output based on: a single syntax element indicating the signaling mode of the set of output layers, a second image belonging to the same block access that the first image indicated as being output, a second image belonging to a higher layer than the first image, and a second image belonging to the output layer. 12. The system of claim 8, wherein the single syntax element has a value of 2 to indicate the signaling mode of the set of output layers. 13. The system of claim 8, wherein the set of parameters is a set of video parameters. 14. The system of claim 8, wherein the determination code is configured to cause the at least one processor to determine that the first image should not be output based on: a single syntax element indicating the signaling mode of the output layer set, a parameter set identifier value greater than 0, and a second image belonging to the same access block as the first image, indicated as not belonging to any output layer. 15. A non-transitory computer-readable medium storing computer instructions configured to cause at least one processor to implement the method of any one of claims 1 to 7.

Images