RU2779911C1 - Signaling of scalability parameters in the video bitstream - Google Patents

Abstract

FIELD: video technology.

SUBSTANCE: group of inventions relates to video encoding/decoding technologies, or rather to signaling scalability in terms of space, quality and multiple angles in the video bitstream. A method for determining the output level for inter-level prediction using at least one processor is proposed. According to the method, an encoded video sequence is obtained from an encoded video bitstream. Next, the first syntactic element is obtained indicating the number of levels included in the encoded video sequence; the second syntactic element is obtained indicating whether all levels included in the encoded video sequence are independent; the third syntactic element is obtained indicating the output mode based on the first syntactic element indicating that the number of levels is more than one, and a second syntactic element indicating that not all levels are independent. And also the output mode is determined based on the third syntactic element; one or more output levels from a set of levels are determined based on the output mode that has been determined; and one or more output levels are output.

EFFECT: increase in scalability signaling accuracy with respect to multiple angles in the video bitstream.

20 cl, 11 dwg, 1 tbl

Description

Link to related application

[1] The present invention claims priority of U.S. Provisional Application No. 62/903,603, filed September 20, 2019, and U.S. Ordinary Application No. 17/019,959, filed September 14, 2020, the entire scope of which is incorporated herein.

The technical field to which the invention belongs

[2] Presented in this application, the invention relates to the encoding and decoding of video, and more specifically to signaling scalability in terms of space, quality and multiple angles in the video bitstream.

State of the art

[3] It is known to encode and decode video by inter-picture prediction with motion compensation. Uncompressed digital video may include a sequence of images, each of which has a spatial size of, for example, 1920×1080 luma samples and their associated chrominance samples. The image sequence may have a fixed or variable image rate (informally also referred to as frame rate), such as 60 images per second, or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (resolution of 1920x1080 luma samples at 60Hz frame rate) requires about 1.5Gbps of bandwidth. An hour of such video requires more than 600 GB of storage.

[4] One goal of video encoding and decoding may be to reduce redundancy in the input video signal by compressing. Compression can help to alleviate the aforementioned bandwidth or storage requirements, in some cases by two orders of magnitude or more. You can use both lossless and lossy compression, as well as a combination of both. Lossless compression refers to techniques for reconstructing an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the discrepancy between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used for video. The amount of distortion allowed depends on the application; for example, users of some custom streaming applications may tolerate higher distortion than users of broadcast television applications. With an achievable compression ratio, a higher allowed/allowable distortion may result in a higher compression ratio.

[5] The video encoder and decoder may use techniques from several broad categories including, for example, motion compensation, transformation, quantization, and entropy coding. Some of these methods will be presented below.

[6] Traditionally, video encoders and decoders worked with a given picture size, which in most cases was determined and remained constant for a coded video sequence (CVS), a group of pictures (GOP), or a similar time period for a group of pictures. For example, in MPEG-2, system decisions change the horizontal resolution (and hence image size) depending on factors such as scene activity, but only for I images, that is, typically for a group of images. Apply resampling of reference pictures for different resolutions in the encoded video sequence, such as in ITU-T Rec. H.263, Appendix P. In this case, the image size does not change, only reference images are recalculated, which potentially affects only the elements of the applied image (in the case of a decrease in resolution) or only on the elements of the processed scene (when the resolution is increased). Further, in H.263, Annex Q allows the resampling of an individual macroblock by halving or halving (on each dimension). Again, the image size remains unchanged. The macroblock size is fixed in H.263 and therefore does not need to be signaled.

[7] Predicted image resizing has become a common trend in modern video coding. For example, VP9 allows you to resample the reference image and change the resolution of the entire image. Similar proposals coming in for VVC (e.g., in Gendry et al. "Adaptive Resolution Resizing (ARC) in VVC", Joint Video Team paper JVET-M0135-v1, Jan 9-19, 2019, incorporated herein in its entirety ) allow resampling the entire reference images to obtain different (higher or lower) resolutions. In this document, it has been proposed to encode the various potential resolutions in a sequence parameter set with reference to them via the syntax elements of each image in the image parameter set.

The essence of the invention

[8] An embodiment provides a method for decoding an encoded video bitstream by at least one processor. The method includes obtaining an encoded video sequence from an encoded video bitstream; obtaining a first syntax element indicating the number of levels included in the encoded video sequence; obtaining a second syntax element indicating whether all layers included in the encoded video sequence are independent; obtaining a third syntax element indicating the output mode based on the first syntax element indicating that there are more than one levels and the second syntax element indicating that not all levels are independent; determining an output mode based on the third syntax element; determining one or more output levels from the plurality of levels based on the output mode that has been determined; and outputting one or more output levels.

[9] An embodiment provides a device for decoding an encoded video bitstream. The device includes at least one memory configured to store program code and at least one processor configured to read the program code and operate in accordance with instructions of the program code. The program code includes a first acquisition code configured to obtain an encoded video sequence from an encoded video bitstream by at least one processor; a second acquisition code configured to receive, by at least one processor, a first syntax element indicating the number of layers included in the encoded video sequence; a third acquisition code configured to receive, by at least one processor, a second syntax element indicating whether all layers included in the encoded video sequence are independent; a fourth obtaining code configured to receive by at least one processor a third syntax element indicating an output mode based on a first syntax element indicating more than one level and a second syntax element indicating that not all levels are independent. The first determination code is configured to determine the output mode by at least one processor based on the third syntax element. The second determination code is configured to determine one or more output levels based on the output mode that has been determined. The output code is configured for at least one processor to output one or more output levels.

[10] An embodiment provides a computer-readable storage medium that stores instructions containing one or more instructions that, when executed by one or more processors of a device for decoding an encoded video bitstream, cause the one or more processors to perform the following operations: obtain an encoded video sequence from encoded video bitstream; obtaining a first syntax element indicating the number of layers included in the encoded video sequence; obtaining a second syntax element indicating whether all layers included in the encoded video sequence are independent; deriving a third syntax element indicating an output mode based on a first syntax element indicating that there are more than one levels and a second syntax element indicating that not all levels are independent; determining an output mode based on the third syntax element; determining one or more output levels from the plurality of levels based on the output mode that has been determined, and outputting the one or more output levels.

Brief description of the drawings

[11] Additional features, nature and various advantages of the disclosed invention will be apparent from the following detailed description and the accompanying drawings, in which:

[12] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system in accordance with an embodiment.

[13] FIG. 2 is a schematic diagram of a simplified block diagram of a communication system in accordance with an embodiment.

[14] FIG. 3 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[15] FIG. 4 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[16] FIG. 5 is a schematic representation of options for ARC/RPR parameter signaling in accordance with an embodiment.

[17] FIG. 6A-6B are examples of syntax tables according to an embodiment.

[18] FIG. 7 is a schematic representation of signaling levels in a VPS according to an embodiment.

[19] FIG. 8 is a schematic diagram of display format list signaling in a VPS according to an embodiment.

[20] FIG. 9 is a schematic representation of display format indicator signaling in a VPS according to an embodiment.

[21] FIG. 10 is a flowchart of an example of a process for decoding an encoded video bitstream according to an embodiment.

[22] FIG. 11 is a schematic representation of a computer system in accordance with an embodiment.

Detailed description

[23] FIG. 1 shows a simplified block diagram of a communication system (100) according to an embodiment of the present invention. The system (100) may include at least two terminals (110, 120) connected by a network (150). In unidirectional data transmission, the first terminal device (UE) may encode the video data in place for transmission to another terminal device (120) via the network (150). The second terminal (120) may receive the encoded video data of another terminal via the network (150), decode the encoded data, and display the recovered video data. One-way communication may be inherent in media service applications and the like.

[24] FIG. 1 shows a second pair of terminals (130, 140) for bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bi-directional data transmission, each terminal (130, 140) may encode video data received locally for transmission to another terminal over the network (150). Each terminal device (130, 140) may also receive the encoded video data transmitted by the other terminal device, decode the encoded data, and display the recovered video data on a local display device.

[25] FIG. 1, terminal devices (110, 120, 130, 140) may be illustrated as servers, personal computers, and smartphones, but this does not limit the principles of the present invention. Embodiments of the present invention find application in laptop computers, tablet computers, media players and/or dedicated video conferencing equipment. Network (150) represents any number of networks that carry encoded video data between terminals (110-140), including, for example, wired and/or wireless networks. The communication network (150) allows data to be exchanged in a circuit-switched and/or packet-switched mode. Exemplary networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For the purposes of this discussion, the architecture and topology of the network (150) may not be relevant to the present invention unless specifically noted below.

[26] FIG. 2 shows, as an example of the application of the disclosed invention, the placement of a video encoder and a video decoder in a streaming environment. The disclosed invention may be equally applicable to other video processing applications, including, for example, video conferencing, digital television, storage of compressed video on digital media, including CD, DVD, memory card, and the like.

[27] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a digital camera, creating, for example, an uncompressed video stream (202). The sample stream (202), shown in bold to highlight the large amount of data compared to the encoded video data, may be processed by an encoder (203) connected to a camera (201). The encoder (203) may include hardware, software, or a combination thereof for providing or implementing aspects of the disclosed invention, as described in more detail below. The encoded video data (204), shown with a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server (205) for future use. One or more streaming clients (206, 208) may access the streaming server (205) to retrieve copies (207, 209) of the encoded video data (204). The client (206) may include a video decoder (210) that decodes an incoming copy of the encoded video data (207) and creates an outgoing video stream (211) that can be rendered on a display (212) or other rendering device (not shown). In some streaming systems, the encoded video data (204, 207, 209) may be encoded according to one video coding/compression standard or another. Examples of these standards include ITU-T Recommendation H.265. A video coding standard is being developed under the informal name "universal video coding" (VVC, Versatile Video Coding). The disclosed invention may be used in the context of VVC.

[28] FIG. 3 may be a functional block diagram of a video decoder (210) according to an embodiment of the present invention.

[29] The receiver (310) may receive one or more encoded video sequences for decoding by the video decoder (210); in the same or another embodiment, one coded video sequence at a time, where the decoding of each coded video sequence is independent of other coded video sequences. The encoded video sequence may be received from a channel (312), which may be a hardware/software link to a storage device where the encoded video data is stored. The receiver (310) may receive encoded video data with other data, such as encoded audio data and/or auxiliary data streams, which may be relayed to their respective utilizing entities (not shown). The receiver (310) may separate the encoded video sequence from other data. To combat network jitter, a buffer memory (315) may be connected between the receiver (310) and the entropy decoder/analyzer (320) (hereinafter "analyzer"). When the receiver (310) receives data from a storage/relay device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (315) may not be needed or may be small. For use in best effort packet networks, such as the Internet, buffer memory (315) may be required, may be relatively large, and may preferably have an adaptive size.

[30] The video decoder (210) may include an analyzer (320) for reconstructing the symbols (321) from the entropy coded video sequence. The categories of these symbols include information used to control the operation of the video decoder (210) and possibly information to control a visualization device such as a display screen (212) that is not an integral part of the decoder but may be connected to it, as shown in fig. 3. The control information for the rendering device(s) may be Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The analyzer (320) may analyze/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be in accordance with a video coding technology or standard and may follow various well-known principles, including variable run length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The analyzer (320) can extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOP, Groups of Pictures), images, sub-images, tiles, slices, blocks, macroblocks, coding units (CU, Coding Units), "bricks" (bricks), transformation units (TU, Transform Units ), prediction units (PU, Prediction Units), etc. A tile can point to a rectangular area of a CU/CTU in a specific column and row of tiles in an image. "Brick" can indicate a rectangular area of CU/CTU rows in a particular tile. A slice may point to one or more image "bricks" that reside in a NAL unit. A sub-image may indicate a rectangular region of one or more slices in an image. The entropy decoder/analyzer can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[31] The analyzer (320) may perform an entropy decoding/analysis operation of the video sequence received from the buffer memory (315) to create symbols (321).

[32] Several different modules may be used for symbol reconstruction (321) depending on the type of encoded video image or parts thereof (eg, inter- and intra-pictures, inter- and intra-block) and other factors. Which modules are used and how can be determined by the subgroup management information extracted from the encoded video sequence by the analyzer (320). The flow of such subgroup management information between the analyzer (320) and multiple modules is not shown hereinafter for simplicity.

[33] In addition to the previously mentioned functional blocks, video decoder 210 can be fundamentally divided into several functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with one another and can, at least partially, be nested within one another. However, for the purposes of describing the disclosed invention, a fundamental subdivision into the following functional modules is appropriate.

[34] The first module is the scaling/inverse transform module (351). The scaling/inverse transform module (351) receives the quantized transform coefficient as well as control information including the transform used, block size, quantization factor, quantization scaling matrices, and so on. as character(s) (321) from the analyzer (320). Such a module can output blocks containing sample values that can be entered into the aggregator (355).

[35] In some cases, the output samples of the scaling/inverse transform block (351) may refer to an intra-coded block; that is, to a block that does not use predicted information from previously reconstructed images, but may use predicted information from previously reconstructed portions of the current image. Such predicted information may be provided by the prediction module (352) within the picture. In some cases, the intra-image prediction module (352) generates a block of the same size and shape as the block to be reconstructed using previously reconstructed environment information retrieved from the current (partially reconstructed) image buffer (358). The aggregator (355) in some cases adds, for each sample, the prediction information generated by the intra prediction module (352) to the output sample information provided by the scaler/inverse transform module (351).

[36] In other cases, the output samples of the scaler/demapper (351) may refer to an externally coded block, possibly with motion compensation. In such a case, the motion compensation prediction module (353) may access the reference picture memory (357) to retrieve samples used for prediction. After motion compensation has been applied to the extracted samples according to the symbols (321) associated with the block, these samples can be added by an aggregator (355) to the output of the scaler/demapper (in this case referred to as residual samples or residual signal) to generate output information. counts. Reference picture memory addresses from where the motion compensation prediction module retrieves predicted samples may be governed by motion vectors available to the motion compensation prediction module in symbol form (321), which may have, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values retrieved from reference picture memory when accurate sub-sample motion vectors, motion vector prediction mechanisms, etc. are used.

[37] The output samples of the aggregator (355) can be applied to various loop filtering methods in the loop filter module (356). The video compression technologies may include deblocking filter technologies under the control of parameters included in the encoded video bitstream and become available to the loop filter module (356) as symbols (321) from the analyzer (320), but may also be responsive to metainformation received during decoding of the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as to respond to previously reconstructed and contour filtered sample values.

[38] The loop filter module (356) may provide a stream of samples that may be fed to the renderer (212) as well as stored in reference picture memory for use in future inter-picture prediction.

[39] Some encoded pictures, when fully reconstructed, can be used as reference pictures for future prediction. For example, when the encoded picture is fully reconstructed and the encoded picture is identified as a reference picture (eg, by analyzer (320)), the current reference picture (358) may become part of the reference picture memory (357), and fresh current picture memory may be re-allocated before reconstruction of the next encoded image.

[40] Video decoder 210 may perform decoding operations according to a predetermined video compression technology, such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax given by the used video compression technology or standard, in the sense that the encoded video sequence may adhere to both the syntax of the video compression technology or standard, according to a document or video compression technology standard, in particular according to the profiles of this document. It may also be necessary for consistency that the complexity of the encoded video sequence remain within the boundaries set by the video compression technology level or standard. In some cases, the levels limit the maximum image size, the maximum frame rate, the maximum sampling rate for reconstruction (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the levels, in some cases, may be further limited by the specifications of the Hypothetical Reference Decoder (HRD) and metadata for managing the HRD buffer signaled in the encoded video sequence.

[41] According to an embodiment, the receiver (310) may receive additional (redundant) data with encoded video. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (210) to correctly decode the data and/or more accurately reconstruct the original video data. The side data may be, for example, levels of improvement in time, space, or signal-to-noise ratio (SNR), redundant slices, redundant images, forward error correction codes, and so on.

[42] FIG. 4 is a functional block diagram of a video encoder (203) according to an embodiment of the present invention.

[43] The video encoder (203) may receive video samples from a video source (201) (which is not part of the video encoder) that may capture video image(s) to be encoded by the video encoder (203).

[44] The video source (201) may provide the source video sequence to be encoded by the video encoder (203) in the form of a digital video sample stream of any suitable bit depth (eg: 8 bit, 10 bit, 12 bit, ...), any color space (eg , VT.601 Y CrCB, RGB, …) and any suitable sampling structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (201) may be a storage device where previously prepared video is stored. In a videoconferencing system, the video source (203) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual images that give a sense of movement when viewed in sequence. The images themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc. used. It is not difficult for a person skilled in the art to understand the relationship between pixels and samples. The following description is for samples.

[45] According to an embodiment, the video encoder (203) may encode and compress images of the source video sequence into the encoded video sequence (443) in real time or subject to any other time constraints imposed by the application. Setting the proper coding rate is one of the functions of the controller (450). The controller (450) controls other functional modules as described below and is operatively connected to these modules. The connection is not shown for simplicity. Parameters set by the controller may include parameters related to frequency adjustment (picture skip, quantizer, lambda value applied in rate-distortion optimization, …), image size, group of pictures scheme (GOP, group of pictures), maximum search area motion vectors, etc. A person skilled in the art can easily identify other features of the controller (450) that may be related to the video encoder (203) optimized for a particular system design.

[46] A number of video encoders are configured to operate in a coding loop well known to those skilled in the art. As a very simplified description, an encoding loop may include an encoder coding element (430) ("source encoder") (responsible for creating symbols based on the input image to be encoded and the reference image(s)) and (local ) a decoder (433) built into the video encoder (203) that reconstructs the symbols to produce sample data as a (remote) decoder would (because any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies discussed in the disclosed invention). The reconstructed sample stream enters the reference picture memory (434). Since decoding a stream of symbols results in bit-accurate results independent of the location of the decoder (local or remote), the contents of the reference picture memory will also be the same bit-accurate for the local encoder and the remote encoder. In other words, the predictive part of the encoder "sees" as reference picture samples exactly the same sample values as the decoder would "see" if using prediction during decoding. This fundamental principle of reference picture synchronism (and eventually drift if synchronism cannot be maintained due to, for example, channel errors) is well known to those skilled in the art.

[47] The "local" decoder (433) may operate in the same manner as the "remote" decoder (210) detailed above with reference to FIG. 3. However, again, referring to FIG. 4, since the symbols are available and the entropy encoder (445) and parser (320) can be losslessly encoded/decoded into the encoded video sequence, the entropy decoding parts of the video decoder (210) including channel (312), receiver (310), buffer memory (315) and analyzer (320) may not be fully implemented in the local decoder (433).

[48] It can be noted that any decoding technology present in a decoder, with the exception of analysis/entropy decoding, must also be present in essentially identical functional form in the corresponding encoder. For this reason, the disclosed invention is focused on the operation of the decoder. The description of encoding technologies can be abbreviated as they are the inverse of the decoded technologies described in detail. Only in some areas a more detailed description is required, which is given below.

[49] In operation, the source encoder (430) may perform motion-compensated predictive coding, in which an input picture is predictively encoded based on one or more previously encoded pictures from the video sequence, referred to as "reference pictures". Thus, the encoding engine (432) encodes the differences between the pixel blocks of the input image and the pixel blocks of the reference image(s) that can be selected as the predicted input image reference(s).

[50] The local video decoder (433) may decode the encoded video data of pictures that may be specified as reference pictures based on the symbols generated by the source encoder (430). The operations of the encoding engine (432) may be predominantly lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (433) duplicates the decoding processes that may be performed by the video decoder on the reference pictures and may cause the reconstructed reference pictures to be stored in the reference picture cache (434). Thus, the video encoder (203) can locally store copies of the reconstructed reference pictures having the same content as the reconstructed reference pictures to be received by the video decoder at the receiver side (in the absence of transmission errors).

[51] The predictor (435) may perform prediction searches for the encoding engine (432). Thus, for a new picture to be encoded, the predictor (435) can search the reference picture memory (634) for sample data (as candidates for the role of reference pixel blocks) or some metadata, for example, motion vectors of the reference picture, block shapes etc., which can serve as a proper reference for predicting new images. The predictor (435) may operate on a sample-block-pixel-block basis in order to find appropriate references for prediction. In some cases, according to the search results obtained by the predictor (435), the input image may have prediction references extracted from a plurality of reference pictures stored in the reference picture memory (434).

[52] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[53] The output signal of all the above functional modules can be entropy encoded in an entropy encoder (445). The entropy encoder translates symbols generated by various functional modules into an encoded video sequence by lossless symbol compression according to techniques known to one skilled in the art, such as Huffman coding, variable run length coding, arithmetic coding, etc.

[54] The transmitter (440) may buffer the encoded video sequence generated by the entropy encoder (445) in preparation for transmission over a communication channel (460), which may be a hardware/software link to a storage device where the encoded video data is stored. The transmitter (440) may combine the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (sources not shown).

[55] The controller (450) may control the operation of the video encoder (203). During encoding, the controller (450) may assign to each encoded picture one or another encoded picture type, which may determine the encoding methods applicable to the corresponding picture. For example, images can often be assigned the following image types:

[56] An intra-picture (I-picture) that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, pictures in the Independent Decoder Refresh ("IDR" format). One skilled in the art will be aware of the I-image varieties and their respective applications and features.

[57] A predicted picture (P-picture) that can be encoded and decoded using intra prediction or inter prediction using at most one motion vector and a reference index to predict sample values of each block.

[58] A bi-directionally predicted picture (B-picture) that can be encoded and decoded using intra prediction or inter prediction using no more than two motion vectors and reference indices to predict sample values of each block. Likewise, multi-predicted pictures may use more than two reference pictures and associated metadata to reconstruct a single block.

[59] Source images typically allow for spatial division into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coding on a block-by-block basis. Blocks may be predictively encoded with reference to other (previously encoded) blocks determined by the coding assignment applied to the respective images of those blocks. For example, I-picture blocks may be encoded without prediction or predictively with reference to previously encoded blocks of the same picture (spatial prediction or intra prediction). P-picture pixel blocks may be encoded without prediction, either by spatial prediction or temporal prediction with reference to one previously encoded reference picture. Blocks of B-pictures may be encoded without prediction, by spatial prediction or temporal prediction with reference to one or two previously encoded reference pictures.

[60] The video encoder (203) may perform encoding operations according to a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (203) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Therefore, the encoded video data may conform to the syntax given by the video coding technology or standard being used.

[61] According to an embodiment, the transmitter (440) may transmit additional data with encoded video. Video encoder (430) may include such data as part of an encoded video sequence. The additional data may include temporal/spatial/SNR enhancement levels, other forms of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, and so on.

[62] Recently, a method of aggregating a compressed area or extracting multiple semantically independent image elements into a single video image has received attention. In particular, for example, in the context of 360° encoding or certain video surveillance applications, multiple semantically independent source images (for example, a six-cube surface of a 360° cube scene, or incoming camera images in a multi-camera surveillance setup) may require separate adaptive resolution settings for processing. different scenes at a certain point in time. In other words, encoders at a given point in time can choose different resampling factors for different semantically independent images that make up an entire video surveillance scene or a 360° scene. When assembled into a single picture, this in turn requires the resampling of the reference picture and the availability of adaptive resolution coding signaling for portions of the encoded picture.

[63] The following are terms that will be referred to in the remainder of this document.

[64] In some cases, a sub-image is a rectangular arrangement of samples, blocks, macroblocks, coding units, or similar semantically combined entities that can be encoded on their own in a different resolution. One or more sub-images may comprise an image. One or more encoded sub-images may constitute an encoded image. One or more encoded sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In some environments, one or more coded sub-images can be assembled into a compressed region without sample-level recoding in the coded picture, and in similar or other cases, one or more coded sub-images can be extracted from the coded picture in the compressed region.

[65] Reference Picture Resampling (RPR) or Adaptive Resolution Change (ARC) refers to mechanisms for changing the resolution of a picture or sub-picture in a coded video sequence, for example, by resampling a reference picture. In the following, the RPR/ARC parameters denote control information that is needed to change the adaptive resolution, and which may include, for example, filter parameters, scaling factors, output and/or reference picture resolutions, various control flags, and so on.

[66] In embodiments, encoding and decoding may be performed on a single semantically independent encoded video image. Before describing the essence of encoding/decoding multiple sub-pictures with independent RPR/ARC parameters and hidden additional complexity, RPR/ARC parameter signaling options will be introduced.

[67] FIG. 5 shows several embodiments for RPR/ARC parameter signaling. Each embodiment has certain advantages and disadvantages in terms of coding efficiency, complexity, and architecture. For RPR/ARC parameter signaling, a video coding standard or technology may choose one or more such embodiments or options from the prior art. The embodiments must be compatible, and in principle they can be interchanged based on the needs of the application, the standards of the applied technology, or the choice of encoder.

[68] RPR/ARC parameter classes may include:

[69] - x and y resolution up/down factors individually or in combination

[70] - resolution up/down ratios with the addition of a temporal dimension indicating an increase/decrease in a series of images at a constant rate

[71] - any of the above groups of coefficients may include encoding one or more supposedly short syntactic elements that may indicate a table containing the coefficient(s).

[72] - resolution in X and Y coordinates, sample units, blocks, macroblocks, coding units (CU), or any other suitable element of the input/output picture, reference picture, encoded picture, individually or in combination. When there are multiple resolutions (eg, one resolution for the input image, another for the reference image), in some cases one set of values may be derived from another set of values. This can be controlled, for example, using flags. The following is a more detailed example.

[73] - "warped" coordinates, like those used in H.263 Annex P, again in suitable detail as presented above. H.263 Annex P defines one efficient way to encode warped coordinates, but other potentially more efficient ways are also being developed. For example, reversible variable length, where the Huffman encoding of the warped Annex P coordinates is replaced by a binary encoding of an appropriate length, where the length of the binary codeword can be derived, for example, from the maximum image size, possibly multiplied by a certain factor and offset by a certain value, so that the "warp" occurred outside the maximum image size.

[74] - parameters of filters for increasing/decreasing resolution. In embodiments, a single filter is applied to up/down the resolution. However, in embodiments, it is more efficient to give flexibility to the filter, which may require filter parameter signaling. Such parameters can be selected through the index of the list of possible filter designs. The filter may be fully specified (eg, through a list of filter coefficients and suitable entropy coding methods), the filter may be implicitly selected through upscaling/downsampling factors signaled by any of the above mechanisms, and so on.

[75] The following description assumes encoding a fixed number of upscaling/downscaling factors (the same factor can be applied in X and Y coordinates) denoted by a codeword. This codeword may advantageously be encoded with a variable length, eg, with an Ext-Golomb code specific to certain syntax elements in the video coding characteristics, eg, in H.264 and H.265. A suitable scaling of the upscaling/downscaling factors can be selected, for example, according to Table 1:

[76] Many of these mappings can be modeled according to the needs of the application and the capabilities of resolution up/down mechanisms in a video compression technology or standard. Other values may be presented in the table. Values may also be represented by entropy encoding mechanisms that do not include Ext-Golomb codes, such as binary encoding. Such mechanisms may have certain advantages when considering resampling factors outside of the video processing means (mainly the encoder and decoder), for example, via MANE. It should be noted that for cases where a resolution change is not required, you can choose the short code Ext-Golomb. In the table above, this code is represented by one bit. For the most typical case, such a code is more efficient than binary codes.

[77] The number of entries in the table, as well as their semantics, can be configured, in whole or in part. For example, the table base can be specified by a high-level parameter set, such as a sequence or decoder parameter set. In embodiments, one or more such tables may be defined in a video coding technology or standard. The tables may also be selected, for example, by means of a set of decoder or sequence parameters.

[78] Next, we describe a method for incorporating an upscaling/downscaling factor (ARC data) encoded by the above technique into a video coding technology or standard syntax. Similar techniques may be applied to one or more codewords that control the resolution up/down filters. The following are controversial situations where relatively large amounts of data are needed for a filter or other data structures.

[79] FIG. 5A shows the inclusion of ARC data (502) by the H.263 Annex P standard as four warped coordinates in the image header (501), more specifically in the H.263 PLUSPTYPE header extension (503). The choice of such a structure is justified by a) the presence of an image header and b) the expectation of frequent changes in the ARC data. However, overhead can be quite large with H.263 signaling, and scaling factors can be out of picture bounds because the picture header may be temporary.

[80] In the same or other embodiments, ARC parameter signaling may follow the example detailed in FIG. 6A-6B. Fig. 6A-6B depict syntax diagrams in display type by means of notation, roughly within the scope of C programming, which has been used, for example, in video coding standards since at least 1993. Lines in bold indicate syntax elements in the bitstream, normal lines often indicate control flow or setting variables.

[81] Referring to FIG. 6A, a tilegroup header (601) in the form of an exemplary header syntax structure applied to a (possibly rectangular) sub-image may conditionally include a variable length, the Exp-Golomb (602) encoded dec_pic_size_idx syntax element (depicted in bold). The presence of such a syntax element in the header of a group of tiles can be controlled using adaptive resolution (603) - here the flag value is not displayed in bold. This means that the flag is at a point in the bitstream according to the syntax scheme. It is possible to signal in any high-level syntax structure, inside or outside of the bitstream, whether adaptive resolution is applied to this image or parts of it. In the example shown, the signaling is done in a sequence parameter set as shown below.

[82] FIG. 6B shows a fragment of the sequence parameter set (610). The first syntax element presented is the variable adaptive_pic_resolution_change_flag (611). If this element is present, the flag may indicate the application of an adaptive resolution, which in turn may require certain control data. In the example, such control data is conditionally present based on the value of the flag and the if() statement in the parameter set (612) and the tile group header (601).

[83] When using adaptive resolution, this example encodes the output resolution in units of samples (613). Position 613 refers to the parameters output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can determine the output image resolution. In other situations, any value can be subject to certain restrictions in the video coding technology or standard. For example, the level definition may limit the number of output samples that can be the product of the values of these two syntax elements. Also, certain video coding technologies or standards, or external technologies or standards, such as system standards, may restrict the quantitative range (for example, one or more dimensions must be divisible by a power of two) or aspect ratio (for example, width and height must be in the ratio of 4 :3 or 16:9). Such restrictions, well known in the art, may be introduced to reduce the load on hardware devices or for other reasons.

[84] In some applications, it may be recommended that the decoder use a specific reference picture size at the command of the encoder, rather than accepting the default output picture size. In this example, the syntax element reference_pic_size_present_flag (614) controls the reference picture reference sizes (615) (again, the number refers to the width and height).

[85] Finally, a table of possible values for the width and height of the decoded image is shown. Such a table may be represented, for example, by a table index (num_dec_pic_size_in_luma_samples_minus1) (616). The value "minus1" may indicate an interpretation of the meaning of this syntax element. For example, if the encoded value is zero, then there will be one table entry available. If the value is five, then the table will have six records. Next, for each "row" of the table, the width and height of the decoded image is entered into the syntax (617).

[86] The rendered table entries (617) may be indexed by the dec_pic_size_idx (602) syntax element in the tile group header, thereby specifying different decoding sizes (essentially scaling factors) for each tile group.

[87] Certain video coding technologies or standards, such as VP9, support spatial scaling through certain forms of reference picture resampling (signaled differently by the disclosed invention) in conjunction with temporal scaling to enable spatial scaling. In particular, certain reference images can be upsampled using ARC techniques to form a spatial enhancement layer base. These higher resolution images can be enhanced with standard high resolution prediction engines to add detail.

[88] The embodiments presented herein can be used in such an environment. In some cases, in the same or another embodiment, a NAL unit header value, such as a temporal ID field, can be used to indicate not only the temporal but also the spatial level. This approach has certain advantages for certain system designs: for example, with existing Selected Forwarding Units (SFUs) created and optimized for temporal selective forwarding based on the NAL unit header, the temporal ID value can be used unchanged for scalable environments. This requires scaling between the encoded picture size and the temporal level indicated by the temporal ID field in the NAL unit header.

[89] With Reference Picture Resampling (RPR) or Adaptive Resolution Resampling (ARC), a minor change in High Level Syntax (HLS) can become an additional burden to support scalability. In embodiments, inter-layer prediction is performed in a scalable system to improve the enhancement layer coding efficiency. In addition to the motion compensated spatial and temporal predictions available in a single layer codec, interlayer prediction can use resampling video data of the reconstructed reference picture from the reference layer to predict the current enhancement layer. Further, the resampling process for interlayer prediction can be performed at the block level by modifying the existing interpolation process for motion compensation. In embodiments, an additional resampling process to support scaling may not be needed. Embodiments refer to high level syntax elements for supporting space/quality scaling by RPR.

[90] In the same or another embodiment, referring to FIG. 7, the syntax element vps_max_layers_minus1 (703) plus 1 may denote the maximum allowed number of layers in each CVS with reference to the VPS (701). The variable vps_all_independent_layers_flag (704) equal to one may mean that all layers in CVS are encoded independently without applying prediction between layers. The variable vps_all_independent_layers_flag (704) equal to zero may mean that one or more layers in CVS can apply interlayer prediction. If this condition is not met, the vps_all_independent_layers_flag value can be considered equal to one. When the variable vps_all_independent_layers_flag is equal to one, the value of the variable vps_independent_layer_flag[i] (706) can be considered equal to one. When the variable vps_all_independent_layers_flag is zero, the value of vps_independent_layer_flag[0] can be considered equal to one.

[91] Referring to FIG. 7, the variable vps_independent_layer_flag[i] (706) equal to one may mean that the layer with pointer i does not use interlayer prediction. A vps_independent_layer_flag[i] variable of zero may mean that the layer with the i pointer can apply inter-layer prediction, and that the vps_layer_dependency_flag[i] variable is present in the VPS. The variable vps_direct_dependency_flag[i][j] (707) equal to zero may mean that the level with pointer j is not a direct reference level for the level with pointer i. The variable vps_direct_dependency_flag[i][j] equal to one can mean that the level with pointer j is a direct reference level for the level with pointer i. When the variable vps_direct_dependency_flag[i][j] is not present for i and j in the range 0 to vps_max_layers_minus1 inclusive, then its value can be assumed to be zero. The variable DirectDependentLayerIdx[i][j], denoting the direct dependent layer j of layer i, can be obtained as follows:

for(i=1; i<vps_max_layers_minus1; i--)

if(!vps_independent_layer_flag[i])

for(j=i, k=0; j>=0; j--)

if(vps_direct_dependency_flag[i][j])

DirectDependentLayerIdx[i][k++]=j

[92] The GeneralLayerIdx[i] variable indicating the layer index with the nuhlayerid variable equal to vps_layer_id[i] can be obtained as follows:

for(i=0; i<=vps_max_layers_minus1; i++)

GeneralLayerIdx[vps_layer_id[i]]=i

[93] In the same or another embodiment, referring to FIG. 7, when vps_max_layers_minus1 is greater than zero and vps_all_independent_layers_flag is zero, vps_output_layers_mode and vps_output_layer_flags[i] are signalled. The variable vps_output_layers_mode (708) equal to zero can mean that only the highest layer is the output. The variable vps_output_layers_mode equal to one can mean that all levels are output. A vps_output_layers_mode variable of two can mean that all output levels are levels with vps_output_layer_flag[i] (709) set to one. The value of the vps_output_layers_mode variable can range from 0 to 2 inclusive. The value 3 of vps_output_layers_mode is reserved for use by ITU-T | ISO/IEC in the future. If this condition is not met, the vps_output_layers_mode value can be considered equal to one. Variable vps_output_layer_flag[i] equal to one may mean that layer i is to be output. Variable vps_output_layer_flag[i] equal to zero may mean that level i is not to be output. The list OutputLayerFlag[i], where the value 1 can mean that level i is to be output, and the value 0 can mean that level i is not an output, can be obtained as follows:

OutputLayerFlag[vps_max_layers_minus1]=1

for(i=0; i<vps_max_layers_minus1; i++)

if(vps_output_layers_mode==0)

OutputLayerFlag[i]=0

else if(vps_output_layers_mode==1)

OutputLayerFlag[i]=1

else if(vps_output_layers_mode==2)

OutputLayerFlag[i]=vps_output_layer_flag[i]

[94] In the same or another embodiment, referring to FIG. 8, the variable vps_num_rep_formats_minus1 plus 1 may indicate the number of rep_format() syntax structures in the VPS. The vps_num_rep_formats_minus1 value can range from 0 to 255 inclusive. The vps_rep_format_idx[i] variable specifies the pointer (in the rep_format() list of syntax structures in the VPS) of the syntax structure applied to the layer with the GeneralLayerIdx[nuh_layer_id] variable equal to i. The vps_rep_format_idx[i] value can range from 0 to vps_num_rep_formats_minus1 inclusive. The number of bits used to map vps_rep_format_idx[i] is Ceil(Log2(vps_num_rep_formats_minus1+1)). Переменные pic_width_max_vps_in_luma_samples, pic_height_max_vps_in_luma_samples, chroma_format_vps_idc, separate_colour_plane_vps_flag, bit_depth_vps_luma_minus8 и bit_depth_vps_chroma_minus8 применяются для получения значений синтаксических элементов SPS pic_width_max_in_luma_samples, pic_height_max_in_luma_samples, chroma_format_idc, separate_colour_plane_flag, bit_depth_luma_minus8 и bit_depth_chroma_minus8 соответственно для каждого SPS, относящегося к VPS. The variable pic_width_max_vps_in_luma_samples cannot be equal to 0 and can be an integer multiple of max(8, MinCbSizeY). The variable pic_height_max_vps_in_luma_samples cannot be equal to 0 and can be an integer multiple of max(8, MinCbSizeY). The value of the chroma_format_vps_idc variable can range from 0 to 3 inclusive. The bit_depth_vps_chroma_minus8 variable can range from 0 to 8 inclusive.

[95] The subpics_vps_present_flag variable equal to one indicates that the subpicture parameters are present in the VPS RBSP syntax. The subpics_present_flag variable equal to zero indicates that the subpicture parameters are not present in the VPS RBSP syntax. Переменные syntax, max_subpics_vps_minus1, subpic_grid_vps_col_width_minus1, subpic_grid_vps_row_height_minus1, subpic_grid_vps_row_height_minus1, subpic_grid_vps_idx[i][j], subpic_treated_as_pic_vps_flag[i] и loop_filter_across_subpic_enabled_vps_flag[i] применяются для получения значений элементов синтаксиса SPS max_subpics_minus1, subpic_grid_col_width_minus1 subpic_grid_row_height_minus1, subpic_grid_row_height_minus1, subpic_grid_idx[i][j], subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] respectively for each VPS related SPS.

[96] In the same or another embodiment, referring to FIG. 9, the variable use_rep_format_flag equal to one may indicate the presence of the variable sps_rep_format_idx. Переменная use_rep_format_flag равная нулю может означать отсутствие переменной sps_rep_format_idx и присутствие следующих переменных в SPS: pic_width_max_in_luma_samples, pic_height_max_in_luma_samples, chroma_format_idc, separate_colour_plane_flag, bit_depth_luma_minus8, bit_depth_chroma_minus8, max_subpics_minus1, subpic_grid_col_width_minus1 subpic_grid_row_height_minus1, subpic_grid_row_height_minus1, subpic_grid_idx[i][j], subpic_treated_as_pic_flag[i] и loop_filter_across_subpic_enabled_flag[ i]. The sps_rep_format_idx variable can mean that the sps_rep_format_idx-th rep_format() syntax structures in the referenced VPS apply to the levels related to that SPS. При наличии переменные pic_width_max_in_luma_samples, pic_height_max_in_luma_samples, chroma_format_idc, separate_colour_plane_flag, bit_depth_luma_minus8, bit_depth_chroma_minus8, max_subpics_minus1, subpic_grid_col_width_minus1 subpic_grid_row_height_minus1, subpic_grid_row_height_minus1, subpic_grid_idx[i][j], subpic_treated_as_pic_flag[i] и loop_filter_across_subpic_enabled_flag[i] предполагаются равными значениям pic_width_max_vps_in_luma_samples, pic_height_max_vps_in_luma_samples, chroma_format_vps_idc, separate_colour _plane_vps_flag , bit_depth_vps_luma_minus8, bit_depth_vps_chroma_minus8, max_subpics_vps_minus1, subpic_grid_vps_col_width_minus1 subpic_grid_vps_row_height_minus1, subpic_grid_vps_row_height_minus1, subpic_grid_vps_idx[i][j], subpic_treated_as_pic_vps_flag[i] и loop_filter_across_subpic_enabled_vps_flag[i] в синтаксических структурах sps_rep_format_idx-th rep_format() в ссылочном VPS. The subpics_present_flag variable equal to one indicates that the subpicture parameters are present in the RBSP SPS syntax. The subpics_present_flag variable equal to zero indicates that the subpicture parameters are not present in the RBSP SPS syntax. The variable max_subpics_minus1 plus 1 can indicate the maximum number of subpics that can be present in CVS. The variable max_subpics_minus1 may range from 0 to 254. The variable subpic_grid_col_width_minus1 plus 1 may indicate the width of each subpicture identification grid element in blocks of four samples. The length of the syntax element is Ceil(Log2(pic_width_max_in_luma_samples / 4)) bits.

[97] The NumSubPicGridCols variable can be obtained as follows: NumSubPicGridCols=(

pic_width_max_in_luma_samples + subpic grid col width minus1*4+3)/

(subpic_grid_col_width_minus1*4+4)

[98] The variable subpic_grid_row_height_minus1 plus 1 may indicate the height of each element of the sub-image identification grid in blocks of four samples. The length of the syntax element is

Ceil(Log2(pic_height_max_in_luma_samples / 4)) bits.

[99] You can get the NumSubPicGridRows variable like this: NumSubPicGridRows=(

pic_height_max_in_luma_samples + subpic_grid_row_height_minus1*4+3) / (subpic_grid_row_height_minus1*4+4)

[100] The variable subpic_grid_idx[i][j] may denote a subpicture pointer of the position of the grid (i, j). The length of the syntax element is

Ceil(Log2(max_subpics_minus1+1)) bits.

[101] Variables SubPicTop[subpic_grid_idx[i][j]],

SubPicLeft[subpic_grid_idx[i][j]], SubPicWidth[subpic_grid_idx[i][j]],

SubPicHeight[subpic_grid_idx[i][j]], and NumSubPics can be obtained like this:

NumSubPics=0

for(i=0; i.<NumSubPicGridRows; i++) {

for(j=0; j<NumSubPicGridCols; j++) {

if(i=0)

SubPicTop[subpic_grid_idx[i][j]]=0

else if(subpic_grid_idx[i][j] !=subpic_grid_idx[i-1][j]) {

SubPicTop[subpic_grid_idx[i][j]]=i

SubPicHeight[subpic_grid_idx[i-1][j]]=

i - SubPicTop[subpic_grid_idx[i-1][j]]

}

if(j=0)

SubPicLeft[subpic_grid_idx[i][j]]=0

else if (subpic_grid_idxf i][j] !=subpic_grid_idx[i][j-1]) {

SubPicLeft[subpic_grid_idx[i][j]]=j

SubPicWidth[subpic_grid_idx[i][j]]=

j - SubPicLeft[subpic_grid_idx[i][j-1]]

}

if (i==NumSubPicGridRows-1)

SubPicHeightf subpic_grid_idx[i][j]]=

i - SubPicTop[subpic_grid_idx[i-1][j]]+1

if (j=NumSubPicGridRows-1)

SubPicWidth[subpic_grid_idx[i][j]]=

j - SubPicLeft[subpic_grid_idx[i][j-1]]+1

if(subpic_grid_idx[i][j]>NumSubPics)

NumSubPics=subpic_grid_idx[i][j]

}

}

[102] The variable subpic_treated_as_pic_flag[i] equal to one may indicate that subpicture i of each encoded picture in CVS is treated as a picture in a decoding process that excludes edge filtering operations. The variable subpic_treated_as_pic_flag[i] equal to zero may indicate that subpicture i of each encoded picture in CVS is not treated as a picture in a decoding process that excludes edge filtering operations. If this condition is not met, the value of the subpic_treated_as_pic_flag[i] variable can be considered equal to zero.

[103] The variable loop_filter_across_subpic_enabled_flag[i] equal to one may indicate that loop filtering operations can be performed on the boundaries of sub-image i of each encoded picture in CVS. The variable loop_filter_across_subpic_enabled_flag[i] equal to zero may indicate that loop filtering operations are not performed on the boundaries of subpicture i of each encoded picture in CVS. If this condition is not met, the value of the loop_filter_across_subpic_enabled_pic_flag[i] variable can be considered equal to one.

[104] FIG. 10 is a flow diagram of an example process 1000 for decoding an encoded video bitstream. In some embodiments, one or more of the processing units of FIG. 10 may be performed by decoder 210. In some embodiments, one or more processing units of FIG. 10 can be performed by another device or group of devices without or with decoder 210, such as encoder 203.

[105] Referring to FIG. 10, process 1000 may include deriving an encoded video sequence from an encoded video bitstream (block 1001).

[106] Further, referring to FIG. 10, process 1000 may include obtaining a first syntax element indicating the number of layers included in the coded video sequence and a second syntax element indicating whether all levels included in the coded video sequence are independent layers (block 1002).

[107] Further, referring to FIG. 10, process 1000 may include the following determination, based on the first syntax element, whether the number of levels included in CVS is greater than one level (block 1003).

[108] Further, referring to FIG. 10, based on the first syntax element indicating that the number of levels included in CVS does not exceed one level (NO at block 1003), process 1000 may proceed to block 1004, in which one level is defined as the output level, and then to block 1010 , in which the output level is to be output.

[109] Further, referring to FIG. 10, based on the first syntax element indicating that the number of levels included in CVS is greater than one level (YES at block 1003), process 1000 may proceed to block 1005.

[110] Further, referring to FIG. 10, process 1000 may include determining, based on the second syntax element, whether all layers included in the coded video sequence are independent layers (block 1005).

[111] Further, referring to FIG. 10, process 1000 may proceed to block 1006, in which all levels are defined as output, and then to block 1010, in which all levels are output, based on the second syntax element indicating the independence of all levels included in the encoded video sequence (YES on block 1005).

[112] Further, referring to FIG. 10, based on the second syntax element indicating that not all levels included in the coded

video sequence are independent (NO at block 1003), process 1000 may proceed to block 1007.

[113] Further, referring to FIG. 10, process 1000 may include obtaining a third syntax element indicating an output mode (block 1007).

[114] Further, referring to FIG. 10, process 1000 may include determining an output mode based on the third syntax element (block 1008).

[115] Further, referring to FIG. 10, process 1000 may include determining one or more output levels from a plurality of levels based on an output mode that has been determined (block 1009).

[116] Further, referring to FIG. 10, process 1000 may include outputting one or more output levels (block 1010).

[117] In an embodiment based on the third syntax element indicating that the output mode is the first mode, one or more output levels may include only the highest level of the plurality of levels.

[118] In an embodiment based on the third syntax element indicating that the output mode is the second mode, one or more output levels may include a plurality of levels.

[119] In an embodiment, process 1000 may also include obtaining a fourth syntax element indicating that the corresponding level of the set of levels is to be output, where (based on the third syntax element indicating that the output mode is the third mode) one or more output levels may include this corresponding level.

[120] In an embodiment, the first, second, and third syntax elements may be signaled in a video parameter set (VPS).

[121] In an embodiment, the VPS may also include a fifth syntax element indicating the set of image formats signaled to the VPS, and a sixth syntax element indicating an image format index of the set of image formats to be applied to the layer of the set of levels.

[122] In an embodiment, the VPS may also include a seventh syntax element indicating whether the sub-image parameters are signaled to the VPS.

[123] In an embodiment, a sequence parameter set (SPS) with reference to a VPS may include an eighth syntax element indicating the inclusion of a ninth syntax element in the SPS. The ninth element of the syntax indicates the index of the image format of the set of image formats, signaled in the VPS, which should be applied to the level of the set of levels.

[124] In an embodiment, the SPS may also include a sixth syntax element indicating whether the sub-image parameters are signaled in the SPS.

[125] Although FIG. 10 shows examples of process blocks 1000, in some cases process 1000 may also include additional blocks, fewer blocks, different blocks, or a different block arrangement compared to FIG. 10. Two or more process blocks 1000 may be processed at the same time as an addition or alternative.

[126] Further, the proposed methods can be performed by a processing circuit (eg, one or more processors, as well as one or more integrated circuits). In one example, one or more processors execute a program stored on a computer-readable storage medium to implement one or more of the proposed methods.

[127] The above methods may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, in FIG. 11 shows a computer system 1100 suitable for implementing some embodiments of the disclosed invention.

[128] Computer software may be coded using any suitable machine code or computer language that may be assembled, compiled, linked, or similarly processed to produce code containing instructions that can be executed directly, or through interpretation, execution of microcode, etc. .p., one or more computer central processing units (CPU, central processing units), graphic processors (GPU, Graphics Processing Units), etc.

[129] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[130] The components of the computer system 1100 shown in FIG. 11 are illustrative in nature and are not intended to impose any limitation on the scope of application or functionality of computer software implementing embodiments of the present invention. The component configuration should also not be interpreted as having any dependency or requirement in connection with any one or combination of components shown in the exemplary embodiment of computer system 1100.

[131] The computer system (1100) may include some human interface input devices. Such an input device may be responsible for input by one or more users through, for example, tactile input (e.g., keystrokes, waving, glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), olfactory input (not shown). Interface devices may also be used to capture certain media not necessarily directly related to conscious human input, e.g. sound (e.g. speech, music, external sound), images (e.g. scanned images, photographic images received from a still camera) , video (eg, 2D video, 3D video including stereoscopic video).

[132] The human interface input devices may include one or more of the following (as depicted): keyboard 1101, mouse 1102, touch pad 1103, touch screen 1110 and associated graphics adapter 1150, joystick 1105, microphone 1106, scanner 1107, chamber 1108.

[133] Computer system 1100 may also include some human interface output devices. Such output devices may stimulate the senses of one or more users through, for example, tactile output, sound, light, and smell/taste. Such output devices may include tactile output devices (for example, haptic feedback via the 1110 touch screen, digital glove or 1105 joystick, but may also be haptic feedback devices that do not serve as input devices), audio output devices (for example: speakers 1109, headphones (not shown)), visual output devices (e.g., 1110 screens, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with tactile or no feedback, some of which are capable of 2D visual output or more than 3D output via, for example, stereographic output; virtual reality goggles (not shown), holographic displays and smoke tanks (not shown)) and printers (not shown) .

[134] The computer system 1100 may also include human accessible storage devices and associated media, such as optical media including CD/DVD ROM/RW 1120 with CD/DVD media 1121 and the like, flash cards 1122 memories, 1123 removable hard drive or solid state drive, traditional magnetic media such as tape and floppy disk (not shown), dedicated ROM/ASIC/PLD based devices such as security dongles (not shown), etc. P.

[135] Those skilled in the art should also understand that the term "computer-readable media" as used in connection with the invention disclosed herein does not encompass transmission media, carrier waves, or other short-term signals.

[136] Computer system 1100 may also include an interface to one or more communication networks 1155. Communication networks can be, for example, wireless, wired, optical. Communication networks can be, in addition, local, global, urban, transport and industrial, real-time, latency, etc. Examples of communication networks include local area networks, such as Ethernet, wireless LANs, cellular networks, including GSM, 3G, 4G, 5G, LTE, etc., digital wide area networks of wired or wireless television, including cable television , satellite TV and terrestrial TV, transport and industrial include CANBus, etc. Some networks usually require external network interface adapters (1154) that are connected to some general purpose data ports or peripheral buses (1149) (for example, the USB ports of the computer system 1100; others are usually built into the core of the computer system 1100 by connecting to the system bus, as described below (for example, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system.) Using any of these networks, the computer system 1100 can communicate with other entities. Such communication can be unidirectional with the ability to receive only (for example, broadcast), unidirectional with transmission-only capability (for example, CANbus to some CANbus devices) or bidirectional, for example, to other computer systems using a LAN or WAN. Some protocols and protocol stacks may be used on each of these networks and the network interfaces described above. (1154).

[137] The aforementioned human interface devices, human accessible storage devices, and network interfaces can be connected to the core 1140 of the computer system 1100.

[138] The core 1140 may include one or more central processing units (CPUs) 1141, graphics processing units (GPUs) 1142, dedicated field programmable gate area (FPGA) 1143, hardware accelerators 1144 for some tasks, etc. These devices, together with read-only memory (ROM) 1145, RAM 1146, large capacity internal storage such as user-inaccessible internal hard drives, SSDs, and the like. 1147 may be connected via system bus 1148. In some computer systems, system bus 1148 may be available in the form of one or more physical connectors to allow for expansion with additional CPUs, GPUs, and the like. Peripherals may be connected either directly to the 1148 core system bus or via the 1149 peripheral bus. Peripheral bus architectures include PCI, USB, and the like.

[139] CPU 1141, GPU 1142, FPGA 1143, and accelerators 1144 may execute some instructions that together may constitute the aforementioned computer code. This computer code may be stored in ROM 1145 or RAM 1146. Transient data may also be stored in RAM 1146, while persistent data may be stored in, for example, an internal mass storage 1147. the use of cache memory, which may be closely associated with one or more of the CPU 1141, GPU 1142, mass storage 1147, ROM 1145, RAM 1146, and the like.

[140] Computer-readable media may store computer code for performing various computer-implemented operations. The media and computer code may be specially created for the purposes of the present invention or may be well known and available to those skilled in the art of computer software.

[141] By way of example, and not limitation, a computer system having architecture 1100, and in particular core 1140, can provide functionality through execution by processor(s) (including(s) CPU, GPU, FPGA, accelerators, and etc.) software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the user-accessible mass storage described above, as well as some permanent storage of the kernel 1140, such as internal mass storage 1147 or ROM 1145. implementation of the present invention may be stored in such devices and executed by the core 1140. Computer-readable media may include one or more memory devices or chips, in accordance with specific needs. The software may direct the core 1140, and in particular its processors (including CPUs, GPUs, FPGAs, and the like), to execute specific processes or specific portions of the specific processes described herein, including defining data structures stored in RAM 1146 and modifying such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality through logic hardwired or otherwise embodied in circuitry (e.g., accelerator 1144) that may operate instead of or in conjunction with software to perform specific processes or specific portions of the specific processes described herein. A reference to software may cover logic, and vice versa when appropriate. Reference to computer-readable media may encompass a circuit (eg, an integrated circuit (IC)) where software is stored for execution, a circuit embodying logic for execution, or both, as appropriate. The present invention encompasses any suitable combination of hardware and software.

[142] Although several illustrative embodiments are described here, variations, permutations, and various replacement equivalents are possible and are within the scope of the invention. Thus, those skilled in the art can provide numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the invention and thus fall within its spirit and scope.

Claims (43)

1. A method for determining the output level in inter-layer prediction using at least one processor, in which: obtaining an encoded video sequence from the encoded video bitstream; get the first syntax element indicating the number of levels included in the encoded video sequence; getting a second syntax element indicating whether all layers included in the encoded video sequence are independent; obtaining a third syntax element indicating the output mode based on the first syntax element indicating that the number of levels is greater than one and the second syntax element indicating that not all levels are independent; determining an output mode based on the third syntax element; determining one or more output levels from the plurality of levels based on the output mode that has been determined; and outputting one or more output levels. 2. The method of claim 1, wherein based on the third syntax element indicating that the output mode is the first mode, said one or more output levels includes only the highest level of the plurality of levels. 3. The method of claim 1, wherein, based on the third syntax element indicating that the output mode is the second mode, said one or more output levels includes a plurality of levels. 4. The method of claim 1, further comprising obtaining a fourth syntax element indicating that the corresponding level of the plurality of levels is to be inferred, wherein, based on the third syntax element indicating that the output mode is the third mode, said one or more output levels contains said corresponding level. 5. The method of claim 1, wherein the first, second, and third syntax elements are signaled in a video parameter set (VPS). 6. The method of claim 5, wherein the VPS also includes a fifth syntax element indicating the number of image formats signaled to the VPS and a sixth syntax element indicating an image format index of the plurality of image formats to apply to a layer of the plurality of layers. 7. The method of claim 5, wherein the VPS also includes a seventh syntax element indicating whether the sub-image parameters are signaled to the VPS. 8. The method of claim 5, wherein the Sequence Parameter Set (SPS) referring to the VPS includes an eighth syntax element indicating that the SPS includes a ninth syntax element indicating an index of an image format from a plurality of image formats signaled to the VPS to apply. to a level of multiple levels. 9. The method of claim 8, wherein the SPS also includes a sixth syntax element indicating whether the sub-image parameters are signaled to the SPS. 10. A device for determining the output level for inter-level prediction, comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to the instructions of the program code, wherein the program code includes: a first acquisition code configured to cause at least one processor to obtain an encoded video sequence from the encoded video bitstream; a second acquisition code configured to cause at least one processor to obtain a first syntax element indicative of the number of levels included in the encoded video sequence; a third acquisition code configured to cause at least one processor to obtain a second syntax element indicating whether all layers included in the encoded video sequence are independent; a fourth acquisition code configured to cause at least one processor to obtain a third syntax element indicating an output mode based on a first syntax element indicating that the number of levels is greater than one level and a second syntax element indicating that not all levels are independent; a first determination code configured to cause at least one processor to determine an output mode based on the third syntax element; a second determination code configured to cause at least one processor to determine one or more output levels from the plurality of levels based on the output mode that has been determined; and output code configured to cause at least one processor to output said one or more output levels. 11. The apparatus of claim 10, wherein said one or more output levels includes only the highest level of the plurality of levels based on a third syntax element indicating that the output mode is the first mode. 12. The apparatus of claim 10, wherein the one or more output levels include a plurality of levels based on a third syntax element indicating that the output mode is the second mode. 13. The apparatus of claim 10, wherein the program code also includes a fifth acquisition code for obtaining a fourth syntax element indicating that the corresponding level of the plurality of levels is to be output, wherein said one or more output levels comprises said corresponding level based on the third syntax element indicating that the output mode is the third mode. 14. The apparatus of claim 10, wherein the first, second, and third syntax elements are signaled in a video parameter set (VPS). 15. The apparatus of claim 14, wherein the VPS also includes a fifth syntax element indicating the number of image formats signaled to the VPS and a sixth syntax element indicating an image format index of the plurality of image formats to apply to a layer of the plurality of layers. 16. The apparatus of claim 14, wherein the VPS also includes a seventh syntax element indicating whether the sub-image parameters are signaled to the VPS. 17. The apparatus of claim 14, wherein the Sequence Parameter Set (SPS) referring to the VPS includes an eighth syntax element indicating that the SPS includes a ninth syntax element that indicates an index of an image format from a plurality of image formats signaled to the VPS for application to a level from a plurality of levels. 18. The apparatus of claim 17, wherein the SPS also includes a sixth syntax element indicating whether the sub-image parameters are signaled to the SPS. 19. A computer-readable storage medium that stores instructions containing one or more instructions that, when executed by one or more processors of the device for determining the output level in inter-layer prediction, cause one or more processors to execute: obtaining an encoded video sequence from the encoded video bitstream; obtaining a first syntax element indicating the number of levels included in the encoded video sequence; obtaining a second syntax element indicating whether all layers included in the encoded video sequence are independent; obtaining a third syntax element indicating the output mode, based on the first syntax element indicating that the number of levels is greater than one, and the second syntax element indicating that not all levels are independent; determining an output mode based on the third syntax element; determining one or more output levels from the plurality of levels based on the output mode that has been determined; and outputting one or more output levels. 20. The computer-readable medium of claim 19, wherein the first, second, and third syntax elements are signaled in a video parameter set (VPS).

Images