RU2810966C1 - Methods of signaling combination of reference image resampling and spatial scalability - Google Patents

Abstract

FIELD: video encoding.

SUBSTANCE: first flag is obtained from the encoded video bitstream indicating that resampling of the reference image is enabled for the encoded video sequence (CVS). The value of the second flag indicating whether the image resolution is changed in the CVS is determined. Based on the value of the second flag indicating that the image resolution is changed in the CVS, the CVS is decoded using reference image resampling for spatial scalability and adaptive resolution resampling (ARC). Based on the value of the second flag indicating that the image resolution is not changed in the CVS, the CVS is decoded using reference picture resampling for spatial scalability and without using reference picture resampling for ARC. Based on the value of the second flag indicating that the image resolution is not changed in CVS, a fifth flag included in the image parameter set referencing the SPS indicates that the image resolution is not changed in CVS.

EFFECT: increase of efficiency of image encoding.

15 cl, 34 dwg, 1 tbl

Description

Cross reference to related application

[1] This application claims priority under 35 USC § 119 to US Provisional Application No. 63/027,835 filed May 20, 2020, US Provisional Application No. 63/035,647 filed June 5, 2020, US Provisional Application No. 63 /036,174, filed June 8, 2020, U.S. Provisional Application No. 63/036,342, filed June 8, 2020, and U.S. Application No. 17/211,236, filed March 24, 2021, with the United States Patent and Trademark Office , the contents of which are incorporated herein by reference in their entirety.

Field of technology to which the invention relates

[2] The disclosed subject matter of the invention relates to video encoding and decoding, and more specifically to signaling a combination of reference picture resampling and spatial scalability in a multi-layer video codestream.

State of the art

[3] Video encoding and decoding using inter-frame prediction with motion compensation is known. Uncompressed digital video may consist of a series of images, each image having a spatial dimension, such as 1920 x 1080 luminance samples and associated chrominance samples. A series of images can have a fixed or variable image rate (informally also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 luminance samples resolution at 60 Hz frame rate) requires a bandwidth of close to 1.5 Gbps. An hour of such video requires more than 600 gigabytes (GB) of disk space.

[4] One of the purposes of video encoding and decoding may be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, or a combination of both, can be used. Lossless compression refers to techniques in which an exact copy of the original signal can be reconstructed from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal suitable for the intended application. In the case of video, lossy compression is widely used. The acceptable degree of distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of broadcast television applications. The achievable compression ratio may reflect the following: Higher resolution/acceptable distortion may result in a higher compression ratio.

[5] The video encoder and video decoder may use techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding, some of which will be presented below.

[6] Historically, video encoders and video decoders have tended to operate at a given picture size, which in most cases was defined and remained constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2 it is known that the system design changes the horizontal resolution (and thus the image size) depending on factors such as scene activity, but only for I-pictures, hence typically for GOP. Resampling of reference images to use different resolutions in CVS is known, for example, from ITU-T Rec. H.263, Annex P. However, here the image size is not changed, only the reference images are resampled, resulting in potentially only parts of the image canvas being used (in in the case of downsampling) or capturing only parts of the scene (in the case of upsampling). In addition, H.263 Annex Q allows an individual macroblock to be resampled by a factor of two (in each dimension), up or down. Again, the image size remains the same. The macroblock size is fixed in H.263 and therefore does not need to be signalled.

[7] Image resizing in predicted images has become more common in modern video coding. For example, VP9 allows you to resample a reference image and change the resolution of the entire image. Likewise, some proposals made regarding VVC (including, for example, Hendry, et al., “On adaptive resolution change (ARC) for VVC,” JVT paper JVET-M0135-v1, January 9-19 2019, incorporated herein in its entirety) allow entire reference images to be resampled to a different, higher or lower, resolution. This document proposes to encode different resolution candidates in a sequence parameter set and refer to them using syntax elements for each image in the image parameter set.

Disclosure of the invention

[8] In an embodiment, there is provided a method for decoding a coded video bitstream using at least one processor, comprising: obtaining from the coded video bitstream a first flag indicating that resampling of a reference picture is enabled for a coded video sequence (CVS); determining a value of a second flag indicating whether the image resolution is changed in CVS; based on the value of the second flag indicating that the image resolution is changed in the CVS, decoding the CVS using resampling of the reference image for spatial scalability and adaptive resolution resampling (ARC); and, based on the value of the second flag indicating that the image resolution is not changed in the CVS, decoding the CVS using reference picture resampling for spatial scalability and without using reference picture resampling for ARC.

[9] In an embodiment, there is provided an apparatus for decoding an encoded video bitstream, including at least one memory element configured to store program code; and at least one processor configured to read the program code and operate in accordance with the instructions of the program code, the program code including: a first acquisition code configured to cause the at least one processor to obtain from the encoded video bitstream a first flag indicating that reference image resampling is enabled for coded video sequence (CVS); determination code configured to cause the at least one processor to determine the value of a second flag indicating whether the image resolution is changed in CVS; a first decoding code configured to cause the at least one processor, based on the value of the second flag indicating that the image resolution is changed in the CVS, to decode the CVS using reference image resampling for spatial scalability and adaptive resolution resampling (ARC); and a second decoding code configured to cause the at least one processor, based on the value of the second flag indicating that the image resolution is not changed in the CVS, to decode the CVS using reference picture resampling for spatial scalability and without using reference picture resampling for ARC

[10] In an embodiment, a non-transitory computer-readable medium is provided storing instructions including: one or more instructions that, when executed by one or more processors of an apparatus for decoding an encoded video bitstream, cause one or more processors to: obtain from the encoded video bitstream a first flag, indicating that resampling of the reference picture is enabled for the coded video sequence (CVS); determine the value of a second flag indicating whether the image resolution is changed in CVS; based on the value of the second flag indicating that the image resolution is changed in the CVS, decode the CVS using reference image resampling for spatial scalability and adaptive resolution resampling (ARC); and based on the value of the second flag indicating that the image resolution is not changed in the CVS, decode the CVS using reference picture resampling for spatial scalability and without using reference picture resampling for ARC.

Brief description of drawings

[11] Additional features, spirit and various advantages of the disclosed subject matter will be better understood from the following detailed description and the accompanying drawings, in which the following is shown:

[12] In FIG. 1 is a schematic illustration of a simplified block diagram of a communication system in accordance with an embodiment.

[13] In FIG. 2 is a schematic illustration of a simplified block diagram of a communication system in accordance with an embodiment.

[14] In FIG. 3 is a schematic illustration of a simplified block diagram of a decoder in accordance with an embodiment.

[15] In FIG. 4 is a schematic illustration of a simplified block diagram of an encoder in accordance with an embodiment.

[16] In FIG. 5A-5E show schematic illustrations of ARC parameter signaling options in accordance with an embodiment, in accordance with an embodiment.

[17] In FIG. 6A-6B show schematic illustrations of examples of syntax tables in accordance with an embodiment.

[18] FIG. 7 shows an example of a prediction structure for adaptive resolution scalability according to an embodiment.

[19] FIG. 8 shows an example of a syntax table according to an embodiment.

[20] In FIG. 9 is a schematic illustration of a simplified flowchart of parsing and decoding a POC cycle per access unit and access unit counter value in accordance with an embodiment.

[21] In FIG. 10 is a schematic illustration of a video bitstream structure containing multi-layer sub-images in accordance with an embodiment.

[22] and fig. 11 is a schematic illustration of displaying a selected sub-image with improved resolution in accordance with an embodiment.

[23] In FIG. 12 is a flowchart of a process for decoding and displaying a video bitstream containing multi-layer sub-images in accordance with an embodiment.

[24] In FIG. 13 is a schematic illustration of a 360° video display with an enhanced sub-image layer in accordance with an embodiment.

[25] In FIG. 14 shows an example of arrangement information of a sub-image and its corresponding layer and image prediction structure according to an embodiment.

[26] In FIG. 15 shows an example of sub-image layout information and their corresponding layer and image prediction structure with local area spatial scalability modality in accordance with an embodiment.

[27] In FIG. 16A-16B show examples of syntax tables for sub-image layout information in accordance with embodiments.

[28] In FIG. 17 shows an example of a SEI message syntax table for sub-image layout information according to an embodiment.

[29] In FIG. 18 shows an example syntax table for specifying output layers and profile/tier/layer information for each set of output layers according to an embodiment.

[30] In FIG. 19 shows an example of a syntax table to indicate an output layer mode for each set of output layers according to an embodiment.

[31] In FIG. 20 shows an example syntax table for specifying the current sub-image of each layer for each set of output layers in accordance with an embodiment.

[32] In FIG. Figure 21 shows an example of a raw byte sequence payload (RBSP) syntax table.

[33] In FIG. 22 shows an example syntax table for specifying an output layer set with an output layer set mode.

[34] FIG. 23 shows an example syntax table for indicating reference picture resampling and resolution changes in a coded video sequence in a sequence parameter set (SPS).

[35] In FIG. Figure 24 shows an example syntax table for specifying the image size in PPS (PPS).

[36] In FIG. 25A-25C show flow diagrams of exemplary processes for decoding an encoded video bitstream in accordance with an embodiment.

[37] In FIG. 26 is a schematic illustration of a computer system in accordance with one embodiment.

Carrying out the invention

[38] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. The system (100) may include at least two terminals (110-120) connected to each other via a network (150). For unidirectional data transmission, the first terminal (110) may encode video data at a local location for transmission to another terminal (120) via the network (150). The second terminal (120) may receive encoded video data of another terminal from the network (150), decode the encoded data, and display the reconstructed video data. Unidirectional data transfer may be common in media serving applications and the like.

[39] FIG. 1 shows a second pair of terminals (130, 140) providing support for bi-directional transmission of encoded video, which may occur, for example, during video conferencing. For bidirectional data transmission, each terminal (130, 140) may encode video data captured at a local location for transmission to another terminal via the network (150). Each terminal (130, 140) can also receive encoded video data transmitted by another terminal, can decode the encoded data, and can display the reconstructed video data on a local display device.

[40] In FIG. 1, the terminals (110-140) may be depicted as servers, personal computers, and smartphones, but the principles of the present disclosure cannot be limited to this. Embodiments of the present invention find use in laptop computers, tablet computers, media players, and/or specialized videoconferencing equipment. Network (150) is any number of networks that transmit encoded video data between terminals (110-140), including, for example, wired and/or wireless communication networks. The communications network (150) may communicate over circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For purposes of this discussion, the architecture and topology of the network (150) may not be essential to the operation of the present disclosure unless explained below herein.

[41] FIG. 2 illustrates, as an example of an application for the disclosed subject matter, the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, compressed video storage on digital media including CD, DVD, memory card, etc., and so on. .

[42] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a digital camera, producing, for example, a stream (202) of uncompressed video samples. This sample stream (202), represented by a thick line to highlight the large amount of data compared to the encoded video bitstreams, can be processed by an encoder (203) connected to the camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (204), depicted as a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server (205) for future use. One or more streaming clients (206, 208) may access the streaming server (205) to retrieve copies (207, 209) of the encoded video bitstream (204). The client (206) may include a video decoder (210) that decodes an incoming copy of the encoded video bitstream (207) and creates an outgoing stream (211) of video samples that can be displayed on a display (212) or other rendering device (not shown) . In some streaming systems, the video bitstreams (204, 207, 209) may be encoded according to certain video encoding/compression standards. Examples of these standards include ITU-T Recommendation H.265. A video encoding standard, informally known as Universal Video Coding or VVC, is under development. The disclosed subject matter may be used in the context of VVC.

[43] FIG. 3 may be a functional block diagram of a video decoder (210) according to an embodiment of the present disclosure.

[44] The receiver (310) may receive one or more encoded video sequences to be decoded by the decoder (210); in the same or another embodiment, one encoded video sequence at a time, where the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequence may be received from a channel (312), which may be a hardware/software connection to a storage device in which the encoded video data is stored. The receiver (310) may receive the encoded video data with other data, such as encoded audio data and/or ancillary data streams, which may be sent to their respective using entities (not shown). The receiver (310) can separate the encoded video sequence from other data. To combat network jitter, a buffer memory (315) can be connected between the receiver (310) and the entropy decoder/parser (320) (“parser”). When the receiver (310) receives data from a storage/forwarding device with sufficient bandwidth and controllability or from an iso-synchronous network, the buffer memory (315) may not be needed or may be small. For use in best quality packet networks such as the Internet, a buffer (315) may be required, which may be relatively large and may preferably be of an adaptive size.

[45] The video decoder (210) may include a parser (320) for recovering symbols (321) from the entropy encoded video sequence. Categories of these symbols include information used to control the operation of the decoder (210), and potentially information to control a visualization device, such as a display (212), which is not an integral part of the decoder, but may be connected to it, as has been shown in figure 3. The control information for the rendering device(s) may be in the form of additional enhancement information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser (320) may parse and/or entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be performed in accordance with a video encoding technology or standard and may follow principles well known to those skilled in the art, including variable length encoding, Huffman encoding, arithmetic encoding with or without context sensitivity, and so on. The parser (320) may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include Groups of Pictures (GOPs), Pictures, Tiles, Slices, Bricks, Macroblocks, Coding Tree Units (CTUs), Coding Units (CUs), Blocks, Transform Units (TUs), Prediction Units (PUs), and so on. . A tile can indicate a rectangular CU/CTU area in a specific column and a number of tiles in an image. A brick may indicate a rectangular area of CU/CTU rows within a particular tile. A slice can point to one or more image bricks that are contained in a NAL unit. A sub-image can point to a rectangular area of one or more slices in an image. An entropy decoder/parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[46] The parser (320) may perform an entropy decoding and/or parsing operation on the video sequence received from the buffer (315) to create symbols (321).

[47] Symbol recovery 321 may include a variety of different modules depending on the type of video image or portions thereof (such as inter- and intra-picture, inter- and intrablock) and other factors being encoded. Which modules are enabled and how can be controlled by subgroup control information parsed from the encoded video sequence using the parser (320). The flow of such subgroup control information between the parser (320) and the plurality of modules is not shown below for clarity.

[48] In addition to the functional blocks already mentioned, the decoder 210 can be conceptually divided into a number of functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with each other and can be at least partially integrated into each other. However, for purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into functional modules as follows.

[49] The first module is a scaling and/or inverse transform module (351). The scaling and/or inverse transform module (351) receives the quantized transform factor, as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, and so on, as symbol(s) (321) from parser (320). It can output blocks containing sample values that can be entered into the aggregator (355).

[50] In some cases, the output samples of the scaling and/or demapping module (351) may be related to an intra-encoded block; that is, to a block that does not use prediction information from previously reconstructed images, but may use prediction information from previously reconstructed portions of the current image. Such prediction information may be provided by an intra-frame prediction module (352). In some cases, the intra-frame prediction module (352) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information extracted from the current (partially reconstructed) image (358). The aggregator (355), in some cases, adds for each sample the prediction information generated by the intra-prediction module (352) to the output sample information provided by the scaling and/or inverse transform module (351).

[51] In other cases, the output samples of the scaling and/or inverse transform module (351) may be related to an intercoding and potentially motion-compensated block. In such a case, the motion compensated prediction module (353) may access the reference image memory (357) to retrieve samples used for prediction. After motion compensation of the selected samples in accordance with the symbols (321) associated with the block, these samples can be added by the aggregator (355) to the output of the scaler and/or inverse transform module (here called residual samples or residual signal) to generate information about output samples. The addresses in the reference picture memory from which the motion-compensated prediction module selects prediction samples may be controlled by motion vectors available to the motion-compensated prediction module in the form of symbols (321), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values retrieved from reference image memory, where precise subsample motion vectors, motion vector prediction mechanisms, and so on are used.

[52] The output samples of the aggregator (355) can be subjected to various loop filtering techniques in the loop filter module (356). Video compression technologies may include intra-loop filtering technologies that are controlled by parameters included in the encoded video bitstream and provided to the loop filtering module (356) as symbols (321) from the parser (320), but may also respond to meta-information received during decoding previous (in decoding order) parts of the encoded image or encoded video sequence, and also respond to sample values previously restored and filtered through loop filtering.

[53] The output of the loop filter module (356) may be a stream of samples, which may be output to the renderer (212) and also stored in reference picture memory for use in future inter-frame prediction.

[54] Certain encoded images, after complete restoration, can be used as reference images for future prediction. After the encoded picture has been completely reconstructed and the encoded picture has been identified as a reference picture (eg, by the parser (320)), the current reference picture (358) may become part of the reference picture buffer (357), and fresh current picture memory may be reallocated before starting to restore the next encoded image.

[55] Video decoder 210 may perform decoding operations according to a predetermined video compression technology, which may be documented in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by a video compression technology or standard in use, in the sense that it matches the syntax of the video compression technology or standard as specified in the video compression technology document or standard and, in particular, in its profiles document. It may also be necessary for compliance to ensure that the complexity of the encoded video sequence is within the boundaries defined by the technology level or video compression standard. In some cases, levels limit the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (measured, for example, in megasamples per second), the maximum reference image size, and so on. The limits set by the layers can in some cases be further limited by hypothetical reference decoder (HRD) specifications and HRD buffer management metadata signaled in the encoded video sequence.

[56] In an embodiment, the receiver (310) may receive additional (redundant) coded video data. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (210) to correctly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[57] FIG. 4 may be a functional block diagram of a video encoder (203) according to an embodiment of the present disclosure.

[58] The encoder (203) may receive video samples from a video source (201) (which is not part of the encoder), which may capture video image(s) for encoding by the encoder (203).

[59] The video source (201) may provide the original video sequence for encoding by the encoder (203) in the form of a digital video sample stream, which may be of any suitable bit depth (eg: 8 bits, 10 bits, 12 bits, ...), any color space (eg BT.601 Y CrCB, RGB, ...) and any suitable sample structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In a multimedia serving system, the video source (201) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (203) may be a camera that captures local image information as a video sequence. Video data may be provided as many individual images that, when viewed sequentially, convey motion. The images themselves may be organized as a spatial array of pixels, in which each pixel may contain one or more samples depending on the sample structure used, color space, etc. The relationship between pixels and samples can be readily understood by one skilled in the art. The description below is oriented towards readings.

[60] In an embodiment, the encoder (203) may encode and compress images of the original video sequence into an encoded video sequence (443) in real time or with any other timing constraints as required by the application. Providing an appropriate encoding rate is one of the functions of the controller (450). The controller controls other functional modules as described below and is functionally connected to these modules. The connection is not depicted for clarity. Parameters set by the controller may include parameters related to rate control (image skip, quantizer, lambda value of rate-distortion optimization methods, ...), image size, group of picture (GOP) layout, maximum motion vector search range, and so on . Other functions of the controller (450) can be readily recognized by one skilled in the art as they may relate to a video encoder (203) optimized for a particular system design.

[61] Some video encoders operate in what one skilled in the art would readily recognize as a "coding loop". As a simplified description, the encoding loop may consist of an encoding portion of the encoder 430 (hereinafter referred to as the "source encoder") (responsible for generating symbols based on the input image to be encoded and the reference image(s)), and a (local) decoder (433) built into an encoder (203) that reconstructs the symbols to create sample data that the (remote) decoder can also create (since any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies discussed in the disclosed subject matter ). This reconstructed stream of samples is entered into the reference image memory (434). Since decoding a symbol stream produces bit-accurate results regardless of the location of the decoder (locally or remotely), the contents of the reference picture buffer are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" using prediction during decoding. This fundamental principle of reference picture synchronicity (and the resulting drift if synchronicity cannot be maintained, for example due to channel errors) is well known to one skilled in the art.

[62] The operation of the "local" decoder (433) may be the same as that of the "remote" decoder (210), which has already been described in detail above with reference to FIG. 3. Briefly referring also to FIG. 4, however, since the symbols are available, and encoding and/or decoding of the symbols into the encoded video sequence by the entropy encoder (445) and parser (320) can be performed losslessly, the entropy decoding portion of the decoder (210), including the channel ( 312), sink (310), buffer (315) and parser (320), cannot be fully implemented in the local decoder (433).

[63] At this point, the observation can be made that any decoding technology other than parsing and/or entropy decoding that is present in the decoder must also necessarily be present in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter of the invention focuses on the operation of the decoder. The description of encoding technologies may be shortened because they are the inverse of the fully described decoding technologies. Only some areas require more detailed description, which is given below.

[64] As part of its operation, the source encoder (430) may perform motion-compensated predictive encoding that predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence that have been designated “reference frames.” Thus, the encoding engine 432 encodes the differences between blocks of pixels of the input frame and blocks of pixels of the reference frame(s) that can be selected as prediction reference(s) for the input frame.

[65] The local video decoder (433) may decode encoded frame video data, which may be referred to as reference frames, based on the symbols generated by the source encoder (430). The operations of the encoding engine (432) may preferably be lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can usually be a copy of the original video sequence with some errors. The local video decoder (433) copies the decoding processes that may be performed by the video decoder on the reference frames and may cause the reconstructed reference frames to be stored in a reference picture cache (434). Thus, the encoder (203) may locally store copies of reconstructed reference frames that have common content as reconstructed reference frames that will be received by the far-end video decoder (assuming no transmission errors).

[66] The predictor (435) may perform predictive searches for the encoding engine (432). That is, for a new image to be encoded, the predictor (435) may search the reference image memory (434) for sample data (as reference pixel block candidates) or certain metadata such as reference image motion vectors, block shapes, etc. further, which can serve as a suitable prediction standard for new images. The predictor (435) may operate on a block-by-pixel basis of samples to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction patterns taken from a plurality of reference images stored in the reference image memory (434).

[67] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[68] The output signals of all the above functional modules may be entropy encoded in an entropy encoder (445). The entropy encoder translates the symbols generated by the various functional modules into an encoded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman encoding, variable length encoding, arithmetic encoding, and so on.

[69] The transmitter (440) may buffer the encoded video sequence(s) generated by the entropy encoder (445) to prepare it for transmission through a communication channel (460), which may be a hardware/software connection to a storage device that will store the encoded video data . The transmitter (440) may merge the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or ancillary data streams (sources not shown).

[70] The controller (450) may control the operation of the encoder (203). During encoding, the controller (450) may assign each encoded image a specific encoded image type, which may affect the encoding techniques that can be applied to the corresponding image. For example, images can often be classified into one of the following frame types:

[71] An intra-picture (I-picture) may be one that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, pictures with independent decoder update. One skilled in the art is aware of these variants of I-images and their respective applications and features.

[72] A prediction picture (P picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most one motion vector and a reference index to predict the sample values of each block.

[73] A bidirectional prediction picture (B picture) may be one that can be encoded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Likewise, multi-prediction images can use more than two reference images and associated metadata to reconstruct a single block.

[74] Source images can typically be spatially divided into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. Blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the encoding assignment applied to the corresponding block images. For example, I-picture blocks may be encoded without prediction, or they may be predictively encoded with reference to already encoded blocks of the same image (spatial prediction or intra-prediction). P-picture pixel blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one previously encoded reference picture. B-picture blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one or two previously encoded reference pictures.

[75] The video encoder (203) may perform encoding operations in accordance with a predefined video encoding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (203) may perform various compression operations, including predictive encoding operations, that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore follow the syntax specified by the video encoding technology or standard being used.

[76] In an embodiment, the transmitter (440) may transmit additional data with encoded video. The video encoder (430) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant images and slices, additional enhancement information (SEI) messages, video suitability information (VUI) parameter set fragments, and so on.

[77] Compressed region aggregation, or extracting multiple semantically independent parts of an image into a single video image, has received some attention recently. Particularly in the context of, for example, 360 encoding or certain surveillance applications, multiple semantically independent source images (for example, the surface of six cubes of a cube-projected 360 scene, or individual camera inputs in the case of a multi-camera surveillance setup) may require separate adaptive resolution settings to cope with different activity of each scene at a given time. In other words, coders at a given time may choose to use different resampling factors for different semantically independent images that make up the entire 360-degree scene or surveillance scene. Combining into a single image in turn requires resampling of the reference image and the availability of adaptive resolution encoding signaling for portions of the encoded image.

[78] Below are several terms that are referenced in the remainder of this description.

[79] A sub-image may refer to, in some cases, a rectangular arrangement of samples, blocks, macroblocks, encoding units, or similar objects that are semantically grouped and that can be independently encoded at a modified resolution. One or more sub-images may form an image. One or more coded sub-pictures may form a coded picture. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In certain environments, one or more encoded sub-pictures may be collected in a compressed region without recoding to the sample level into the encoded image, and in the same or other cases, one or more encoded sub-images may be extracted from a coded image in the compressed region.

[80] Adaptive Resolution Changing (ARC) may refer to mechanisms that allow the resolution of an image or sub-image in an encoded video sequence to be changed, for example, by resampling a reference image. In the following, ARC parameters refer to the control information necessary to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, output and/or reference image resolutions, various control flags, and so on.

[81] In embodiments, encoding and decoding may be performed on a single semantically independent encoded video image. Before describing the consequences of encoding/decoding multiple sub-pictures with independent ARC parameters and their implied additional complexity, ARC parameter signaling options must be described.

[82] In FIG. 5A-5E illustrate several embodiments of ARC parameter signaling. As noted for each of the embodiments, they may have certain advantages and certain disadvantages in terms of coding efficiency, complexity and architecture. A video coding standard or technology may select one or more of these embodiments or those known in the art for signaling ARC parameters. The embodiments may not be mutually exclusive and may possibly be interchangeable depending on the needs of the application, the standards used, or the choice of encoder.

[83] ARC parameter classes may include:

[84] - upsampling and/or downsampling factors, separate or combined in the X and Y dimensions,

[85] - upsampling and/or downsampling factors with the addition of a time dimension, indicating an increase/decrease in size at a constant rate for a given number of images,

[86] - either of the above two options may involve encoding one or more presumably short syntax elements that may indicate a table containing the coefficient(s),

[87] Resolution in the X or Y dimension in sample units, blocks, macroblocks, coding units (CU), or any other suitable granularity of the input image, output image, reference image, coded image, in combination or individually. If there is more than one resolution (for example, one for the input image, one for the reference image), then in some cases one set of values can be inferred from another set of values. This can be conveyed, for example, by using flags. See below for a more detailed example.

[88] "deformation" coordinates similar to those used in H.263 Annex P, again with a suitable level of detail as described above. Annex P H.263 specifies one efficient way to encode such deformation coordinates, but other, potentially more efficient ways may also be being developed. For example, reverse Huffman coding with variable length warp coordinates according to Appendix P can be replaced by binary coding of suitable length, where the length of the binary codeword can be, for example, derived from the maximum image size, perhaps multiplied by a certain factor and offset by a certain value, so that take into account “deformation” beyond the limits of the maximum image size.

[89] Upsampling and/or downsampling filter options. In embodiments, there may be only one filter for upsampling and/or downsampling. However, in embodiments, it may be desirable to provide greater flexibility in filter design, and this may require signaling of filter parameters. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g. by a list of filter coefficients using suitable entropy coding techniques), the filter may be implicitly selected by appropriate upsampling and/or downsampling ratios , which in turn are signaled according to any of the mechanisms mentioned above, and so on.

[90] The following description assumes encoding a finite set of upsampling and/or downsampling coefficients (the same coefficient to be used in the X and Y dimensions) specified by a codeword. This codeword may be encoded with a variable length, for example using exponential Golomb code, common to certain syntax elements in video encoding specifications such as H.264 and H.265. One suitable mapping of values for upsampling and/or downsampling factors could, for example, be as shown in Table 1:

TABLE 1 A codeword Exponential Golomb code Source/Target Resolution 0 1 1/1 1 010 1/1.5 (50% increase) 2 011 1.5/1 (50% reduction) 3 00100 1/2 (100% increase) 4 00101 2/1 (100% reduction)

[91] Many similar mappings can be developed according to the needs of the application and the capabilities of the upsampling and downsampling mechanisms available in the video compression technology or standard. The table can be expanded to include more values. Values can also be represented by entropy encoding mechanisms other than Exponential Golomb codes, such as using binary encoding. This may have certain advantages where resampling factors were of interest outside of the video processing engines themselves (primarily the encoder and decoder), for example through MANE. It should be noted that for situations where a change in resolution is not required, the Exponential Golomb code, which is short, can be chosen; there is only one bit in the table above. This may have an advantage in coding efficiency over using binary codes for the most common case.

[92] The number of records in the table, as well as their semantics, can be fully or partially customized. For example, the underlying table structure may be conveyed in a "high" parameter set, such as a sequence or decoder parameter set. In embodiments, one or more such tables may be defined in a video encoding technology or standard and may be selected, for example, by a decoder or sequence parameter set.

[93] The following is a description of how the upsampling and/or downsampling factor (ARC information) encoded as described above can be included in a video encoding technology or standard syntax. Similar considerations may apply to one or more codewords driving the upsampling/downsampling filters. See the discussion below when a filter or other data structures require relatively large amounts of data.

[94] As shown in FIG. 5A, H.263 Annex P includes ARC information (502) in the form of four warp coordinates in the image header (501), specifically in the H.263 PLUSPTYPE header extension (503). This may be a reasonable design choice when a) the image header is available and b) frequent changes to the ARC information are expected. However, the overhead when using H.263 style signaling can be quite high, and the scaling factors may not apply to image boundaries since the image header may be temporary.

[95] As shown in FIG. 5B, JVCET-M135-v1 includes ARC reference information (505) (index) located in the image parameter set (504), index table (506) including target resolutions, which in turn , is inside the sequence parameter set (507). Placing a possible resolution in a table (506) in a sequence parameter set (507) may, in accordance with verbal statements made by the authors, be justified by using the SPS as a point of interoperability negotiation during capability exchange. The resolution can be varied within the limits set by the values in the table (506) from image to image by accessing the appropriate set (504) of image parameters.

[96] Referring to FIG. 5C-5E, the following embodiments may exist for transmitting ARC information in a video bitstream. Each of these options has certain advantages over the embodiments described above. Embodiments may simultaneously be present in the same video coding technology or standard.

[97] In embodiments such as the embodiment of FIG. 5C, ARC information (509) such as a resampling (scaling) factor may be present in a slice header, a GOP header, a tile header, or a tile group header. Fig. 5C illustrates an embodiment that uses a tile group header (508). This may be sufficient in the case of small-sized ARC information, such as a single variable-length codeword ue(v) or a fixed-length codeword of several bits, such as shown above. Having ARC information directly in the tile group header has the added benefit that the ARC information can be applied to a sub-image represented by, for example, that tile group rather than the entire image. See also below. Additionally, even if a video compression technology or standard only provides for changing the adaptive resolution of the entire image (as opposed to, for example, adaptive resolution changes on a per-tile group basis), placing the ARC information in the tile group header instead of placing it in the H. 263 has certain advantages in terms of error tolerance.

[98] In embodiments such as the embodiment of FIG. 5D, the ARC information (512) itself may be present in a corresponding parameter set, such as, for example, an image parameter set, a title parameter set, a tile parameter set, an adaptation parameter set, and so on. Fig. 5D illustrates an embodiment in which an adaptation parameter set (511) is used. The scope of this parameter set may preferably be no larger than an image, such as a group of tiles. The use of ARC information is implemented implicitly by activating the appropriate set of parameters. For example, when a video encoding technology or standard considers only image-based ARC, then an image parameter set or equivalent may be an appropriate option.

[99] In embodiments such as the embodiment of FIG. 5E, ARC reference information (513) may be present in a tile group header (514) or similar data structure. This reference information (513) may refer to a subset of ARC information (515) available in a parameter set (516) with a scope beyond a single image, such as a sequence parameter set or a decoder parameter set.

[100] As shown in FIG. 6A, the tile group header (601), as an exemplary syntactic structure of the header applied to the (possibly rectangular) portion of the image, may conventionally include an exponential Golomb code encoded variable length syntax element dec_pic_size_idx (602 ) (in bold). The presence of this syntax element in a tile group header can be limited by the use of adaptive resolution (603) - here the flag value is not in bold, meaning that the flag is present in the bitstream at the point where it occurs in the syntax diagram. The fact whether adaptive resolution is used for this image or parts thereof can be signaled in any high-level syntactic structure inside or outside the bitstream. In the example shown, this is signaled in the sequence parameter set as shown below.

[101] FIG. 6B also shows a sample of a sequence parameter set (610). The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, this flag may indicate the use of adaptive resolution, which in turn may require certain control information. In the example, such control information is present conditionally based on the value of a flag based on the if() statement in the parameter set (612) and the tile group header (601).

[102] When adaptive resolution is used, this example encodes the output resolution in sample units (613). Item number 613 refers to both output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can determine the resolution of the output image. Elsewhere in the video encoding technology or standard, certain limits may be specified for any value. For example, the level definition may limit the number of total output samples that can be the product of the values of these two syntax elements. Also, certain video encoding technologies or standards, or external technologies or standards, such as system standards, may limit the numbering range (for example, one or both dimensions must be divisible by a power of 2) or the width-to-height ratio (for example, width and height should be in a ratio such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementations or for other reasons, and are well known in the art.

[103] In some applications, it may be desirable for the encoder to instruct the decoder to use a certain reference image size, rather than implicitly assuming that this size is the output image size. In this example, the syntax element reference_pic_size_present_flag (614) conveys the conditional presence of the dimensions (615) of the reference picture (again, the position number refers to both width and height).

[104] Finally, a table of possible decoding image widths and heights is shown. Such a table can be expressed, for example, by specifying the table (num_dec_pic_size_in_luma_samples_minus1) (616). "Minus1" may refer to the interpretation of the meaning of this syntax element. For example, if the encoded value is zero, there is one entry in the table. If the value is five, there are six table entries. Then, for each "row" in the table, the width and height of the decoded image are included in the syntax (617).

[105] The represented table entries (617) may be indexed using the dec_pic_size_idx syntax element (602) in the tile group header, thereby allowing different decoded sizes—essentially, scaling factors—for each tile group.

[106] Some video encoding technologies or standards, such as VP9, support spatial scalability by implementing certain forms of reference image resampling (signaled entirely differently than in the disclosed subject matter) in combination with temporal scalability to provide spatial scalability. In particular, some reference images may be upsampled using ARC-style techniques to a higher resolution to form the base of the spatial enhancement layer. These upsampled images can be enhanced by using normal high-resolution prediction engines to add detail.

[107] The embodiments discussed herein may be used in such an environment. In some cases, in the same or another embodiment, a value in the NAL unit header, such as a temporal identifier field, may be used to indicate not only a temporal layer, but also a spatial layer. This may have certain benefits for certain system configurations; for example, an existing Selected Forwarding Units (SFU) architecture designed and optimized for selected temporal layers forwarded based on the NAL unit header temporary identifier value can be used without modification for scale-out environments. To make this possible, there may be a requirement for a mapping between the encoded image size and the temporal layer indicated by the temporary identifier field in the NAL unit header.

[108] In some video encoding technologies, an access unit (AU) may refer to the encoded image(s), slice(s), tile(s), NAL block(s), and so on, that have been captured and arranged into a corresponding image, slice, tile and/or bitstream of a NAL unit at a given time. Such a point in time could be, for example, the time of composition.

[109] In HEVC and some other video encoding technologies, a picture order counter (POC) value can be used to indicate a selected reference picture among a plurality of reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) contains one or more images, slices or tiles, each image, slice or tile belonging to the same AU may carry the same POC value, from which it can be inferred that it was created from content at the same layout time. In other words, in a scenario where two images/slices/tiles carry the same set POC value, it may indicate that the two images/slices/tiles belong to the same AU and have the same layout time. Conversely, two images/slices/tiles having different POC values may indicate that these images/slices/tiles belong to different AUs and have different composition times.

[110] In embodiments, this tight relationship may be relaxed because the access block may contain images, slices, or tiles with different POC values. By allowing different POC values in the AU, it becomes possible to use the POC value to identify potentially independently decodable pictures/slices/tiles with identical presentation times. This, in turn, can support multiple scalable layers without changing reference picture selection signaling (eg, reference picture set signaling or reference picture list signaling), as described in more detail below.

[111] However, it is still desirable to be able to identify the AU that owns an image/slice/tile with respect to another image/slice/tile having different POC values based only on the POC value. This can be achieved as described below.

[112] In embodiments, an access unit counter (AUC) may be signaled in a high-level syntactic structure such as a NAL unit header, a slice header, a tile group header, an SEI message, a parameter set, or an AU delimiter. The AUC value can be used to identify which NAL units, images, slices or tiles belong to a given AU. The AUC value may correspond to a single point in time of the composition. The AUC value can be a multiple of the POC value. By dividing the POC value by an integer value, the AUC value can be calculated. In some cases, division operations may impose some overhead on decoder implementations. In such cases, slight restrictions in the numbering space of AUC values may allow the division operation to be replaced by shift operations. For example, the AUC value may be equal to the value of the most significant bit (MSB) of the POC value range.

[113] In embodiments, the POC cycle value for each AU (poc_cycle_au) may be signaled in a high-level syntactic structure such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or AU delimiter. The poc_cycle_au value can indicate how many different and consecutive POC values can be associated with the same AU. For example, if the value of poc_cycle_au is 4, images, slices or tiles with a POC value of 0-3 inclusive can be associated with an AU with an AUC value of 0, and images, slices or tiles with a POC value of 4-7 inclusive , can be associated with an AU with an AUC value of 1. Therefore, the AUC value can be derived by dividing the POC value by the poc_cycle_au value.

[114] In embodiments, the poc_cyle_au value may be obtained from information located, for example, in a video parameter set (VPS) that identifies the number of spatial layers or SNR layers in the encoded video sequence. An example of such a possible relationship is briefly described below. Although obtaining as described above may save a few bits in the VPS and therefore may improve encoding efficiency, in some embodiments poc_cycle_au may be explicitly encoded in an appropriate high-level syntactic structure hierarchically below the video parameter set to be able to minimize poc_cycle_au for a given a small portion of a bit stream, such as an image. This optimization can save more bits than can be stored through the acquisition process described above because POC values and/or syntax element values indirectly related to POC can be encoded in low-level syntactic structures.

[115] In embodiments, FIG. 8 shows example syntax tables for signaling a vps_poc_cycle_au syntax element in a VPS (or SPS) that indicates the poc_cycle_au used for all pictures/slices in a coded video sequence, and a slice_poc_cycle_au syntax element that indicates the poc_cycle_au of the current slice in the slice header. If the POC value increases uniformly for each AU, vps_contant_poc_cycle_per_au in the VPS can be set to 1, and vps_poc_cycle_au can be signaled in the VPS. In this case, slice_poc_cycle_au may be signaled implicitly, and the AUC value for each AU may be calculated by dividing the POC value by vps_poc_cycle_au. If the POC value does not increase uniformly per AU, vps_contant_poc_cycle_per_au in the VPS may be set to 0. In this case, vps _access_unit_cnt may not be signaled, while slice_access_unit_cnt may be signaled in the slice header for each slice or image. Each slice or image can have a different slice_access_unit_cnt value. The AUC value for each AU can be calculated by dividing the POC value by slice_poc_cycle_au.

[116] In FIG. 9 is a flowchart illustrating an example of the above-described process. For example, in step S910 the VPS (or SPS) can be analyzed, and in step S920 it can be determined whether the POC cycle on the AU is constant within the encoded video sequence. If the POC cycle on the AU is constant (YES in step S920), then the value of the access unit counter for a specific access unit can be calculated from the poc_cycle_au that is signaled for the encoded video sequence and the POC value of the specific access unit in step S930. If the POC cycle on the AU is not constant (NO in operation S920), then the value of the access unit counter for a specific access unit can be calculated from poc_cycle_au, which is signaled at the picture level, and the POC value of the specific access unit in operation S940. In operation S950, the new VPS (or SPS) can be analyzed.

[117] In embodiments, even though the POC value of an image, slice, or tile may be different, the image, slice, or tile corresponding to AUs with the same AUC value may be associated with the same decoding or output timing. Therefore, without any dependency on interparsing/decoding between images, slices or tiles in the same AU, all or a subset of images, slices or tiles associated with the same AU can be decoded in parallel and can be output to the same point in time.

[118] In embodiments, even though the POC value of an image, slice, or tile may be different, an image, slice, or tile corresponding to an AU with the same AUC value may be associated with the same layout/display time. When the layout time is contained in the container format, even if the images correspond to different AUs, as long as the images have the same layout time, the images can be displayed at the same point in time.

[119] In embodiments, each image, slice, or tile may have the same temporal identifier (temporal_id) in the same AU. All or a subset of images, slices or tiles corresponding to a point in time can be associated with the same time sublayer. In embodiments, each image, slice, or tile may have the same or a different spatial layer identifier (layer_id) in the same AU. All or a subset of images, slices, or tiles corresponding to a point in time can be associated with the same or a different spatial layer.

[120] FIG. 7 shows an example of a video sequence structure with a combination of temporal_id, layer_id, POC and AUC values with adaptive resolution changing. In this example, an image, slice, or tile in the first AU with AUC = 0 may have temporal_id = 0 and layer_id = 0 or 1, while an image, slice, or tile in the second AU with AUC = 1 may have temporal_id = 1 and layer_id = 0 or 1 respectively. The POC value is incremented by 1 for each image, regardless of the temporal_id and layer_id values. In this example, the value of poc_cycle_au may be equal to 2. In embodiments, the value of poc_cycle_au may be set to the number of layers (spatial scalability). Therefore, in this example, the POC value is increased by 2 and the AUC value is increased by 1.

[121] In the above embodiments, all or a subset of the inter-frame or inter-layer prediction structure and reference picture indication may be supported using existing reference picture set (RPS) signaling in HEVC or reference picture list (RPL) signaling. In RPS or RPL, the selected reference picture may be indicated by signaling a POC value or a POC delta value between the current picture and the selected reference picture. In embodiments, RPS and RPL can be used to indicate an inter-picture or inter-layer prediction structure without changing the signaling, but with the following limitations. If the temporal_id value of the reference picture is greater than the temporal_id value of the current picture, the current picture may not use the reference picture for motion compensation or other predictions. If the layer_id value of the reference image is greater than the layer_id value of the current image, the current image may not use the reference image for motion compensation or other predictions.

[122] In embodiments, POC difference-based motion vector scaling for temporal motion vector prediction may be disabled for multiple images in an access unit. Therefore, although each image may have a different POC value in an access unit, the motion vector is not scaled and is used to temporally predict the motion vector in an access unit. This is because a reference picture with a different POC in the same AU is considered a reference picture having the same point in time. Therefore, in an embodiment, the motion vector scaling function may return 1 when the reference image belongs to an AU associated with the current image.

[123] In embodiments, POC difference-based motion vector scaling for temporal motion vector prediction may be optionally disabled for multiple images when the spatial resolution of the reference image differs from the spatial resolution of the current image. When motion vector scaling is enabled, the motion vector is scaled based on both the POC difference and the spatial resolution ratio between the current image and the reference image.

[124] In embodiments, the motion vector may be scaled based on the AUC difference instead of the POC difference to temporally predict the motion vector, especially when poc_cycle_au has a non-uniform value (when vps_contant_poc_cycle_per_au == 0). Otherwise (eg, when vps_contant_poc_cycle_per_au == 1), scaling the motion vector based on the AUC difference may be identical to scaling the motion vector based on the POC difference.

[125] In embodiments, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU (with the same AUC value) with the current image is not scaled based on the AUC difference and is used to predict the motion vector without scaling or with scaling based on the spatial resolution relationship between the current image and the reference image.

[126] In embodiments, the AUC value can be used to identify the boundary of the AU and is used for a hypothetical reference decoder (HRD) operation that requires input and output synchronization to the AU granularity. In embodiments, the decoded image with the highest layer in the AU may be output for display. The AUC value and layer_id value can be used to identify the output image.

[127] In embodiments, an image may include one or more sub-images. Each sub-image can cover a local area or the entire image area. The area supported by a sub-image may or may not overlap with the area supported by another sub-image. The area covered by one or more sub-images may or may not cover the entire image area. If the image includes sub-images, the area supported by the sub-image may be identical to the area supported by the image.

[128] In embodiments, the sub-image may be encoded with an encoding method similar to the encoding method used for the encoded image. The sub-image may be independently encoded or may be encoded depending on another sub-image or encoded image. A sub-image may or may not have any parsing dependency on another sub-image or encoded image.

[129] In an embodiment, the encoded sub-image may be contained in one or more layers. The encoded sub-image in a layer may have a different spatial resolution. The original sub-image may be spatially resampled (eg, upsampled or downsampled), encoded at different spatial resolution parameters, and contained in a bitstream corresponding to the layer.

[130] In embodiments, a sub-image with ( W , H ), where W denotes the width of the sub-image and H denotes the height of the sub-image, respectively, may be encoded and contained in the encoded bitstream corresponding to layer 0, while the sub-image is up (or downsampling from a sub-image with the original spatial resolution, with ( W*S _{w , k} , H* S _{h , k} ), can be encoded and contained in the encoded bitstream corresponding to layer k, where S _{w , k} , S _{h , k} indicate the horizontal and vertical resampling factors. If the values of S _{w , k} , S _{h , k} are greater than 1, the resampling may be upsampling. Whereas, if the values of S _{w , k} , S _{h , k} are less than 1, the resampling may be downsampling.

[131] In embodiments, an encoded sub-image in a layer may have a different visual quality than an encoded sub-image in another layer in the same sub-image or a different sub-image. For example, sub-image i in layer n may be encoded by a quantization parameter Qi _{, n} , while sub-image j in layer m may be encoded by a quantization parameter Qj _{, m} .

[132] In embodiments, an encoded sub-image in a layer can be decoded independently, without any dependency on parsing or decoding, from an encoded sub-image in another layer of the same local area. A sub-image layer that can be independently decoded without reference to another sub-image layer of the same local region may be an independent sub-image layer. An encoded subimage in an independent subimage layer may or may not have a decoding or parsing dependency on a previously encoded subimage in the same subimage layer, but an encoded subimage may not have any dependency on an encoded image in another subimage layer.

[133] In embodiments, an encoded sub-image in a layer may be decoded dependently, with a particular parsing or decoding dependency on an encoded sub-image in another layer of the same local area. A sub-image layer that can be dependently decoded with reference to another sub-image layer of the same local region may be a dependent sub-image layer. An encoded subimage in a dependent subimage may reference an encoded subimage belonging to the same subimage, a previously encoded subimage in the same subimage layer, or both reference subimages.

[134] In embodiments, an encoded sub-image may comprise one or more independent sub-image layers and one or more dependent sub-image layers. However, at least one independent sub-picture may be present for the encoded sub-picture. The layer identifier value (layer_id), which may be present in the NAL unit header or other high-level syntactic structure, of an independent sub-image layer may be 0. A sub-image layer with a layer_id equal to 0 may be the base layer of the sub-image.

[135] In embodiments, an image may include one or more foreground sub-images and one background sub-image. The area supported by the background subimage may be equal to the image area. The region supported by the foreground sub-image may overlap with the region supported by the background sub-image. The background subimage may be a base layer of the subimage, while the foreground subimage may be a non-baseline (enhancement) layer of the subimage. One or more non-base layers of a sub-image may reference the same base layer for decoding. Each non-base subimage layer with layer_id equal to a can reference a non-base subimage layer with layer_id equal to b , where a is greater than b .

[136] In embodiments, an image may include one or more foreground sub-images with or without one background sub-image. Each sub-image may have its own base sub-image layer and one or more non-base (enhancement) layers. Each sub-image base layer may be referenced by one or more non-base sub-image layers. Each non-base subimage layer with layer_id equal to a can reference a non-base subimage layer with layer_id equal to b , where a is greater than b .

[137] In embodiments, an image may include one or more foreground sub-images with or without one background sub-image. Each encoded sub-image in a (base or non-base) sub-image layer may be referenced by one or more non-base layer sub-images that belong to the same sub-image, and one or more non-base layer sub-images that do not belong to the same sub-image. .

[138] In embodiments, an image may include one or more foreground sub-images with or without one background sub-image. The sub-image in layer a can be further divided into multiple sub-images in the same layer. One or more encoded sub-pictures in layer b may reference divided sub-pictures in layer a .

[139] In embodiments, a coded video sequence (CVS) may be a group of coded pictures. The CVS may include one or more coded sub-picture sequences (CSPS), where a CSPS may be a group of coded sub-pictures covering the same local image area. CSPS may have the same or a different temporal resolution as the encoded video sequence.

[140] In embodiments, the CSPS may be encoded and contained in one or more layers. The CSPS may include one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS can reconstruct a sequence of sub-images corresponding to the same local region.

[141] In embodiments, the number of CSPS layers corresponding to a CSPS may be identical to or different from the number of CSPS layers corresponding to another CSPS.

[142] In embodiments, a CSPS layer may have a different temporal resolution (eg, frame rate) than another CSPS layer. The original (uncompressed) subimage sequence may be temporally resampled (eg, upsampled or downsampled), encoded at different temporal resolution parameters, and contained in a bitstream corresponding to the layer.

[143] In embodiments, a sub-image sequence at frame rate F may be encoded and contained in the encoded bitstream corresponding to layer 0, while the sub-image sequence is temporally upsampled (or downsampled) from the original sub-image sequence at F* _{St , k} may be encoded and contained in the encoded bitstream corresponding to layer k _{, where St , k} indicates the time sampling rate for layer k. If the value of _{St , k} is greater than 1, the temporal resampling process may be frame rate upconversion. Whereas, if the value of S _{t , k} is less than 1, the temporal resampling process may be frame rate down-conversion.

[144] In embodiments where sub-images with CSPS layer a are referenced by sub-images with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different based on the spatial resolution of CSPS layer b , the decoded pixels on CSPS layer a are resampled and used as reference. The resampling process can use upsampling or downsampling filtering.

[145] FIG. 10 shows an example video stream including a background CSPS video with a layer_id of 0 and a plurality of foreground CSPS layers. Although the encoded sub-image may include one or more CSPS layers, a background region that does not belong to any foreground CSPS layer may include a base layer. The base layer may contain a background region and a foreground region, while the CSPS enhancement layer may contain a foreground region. The CSPS enhancement layer may have better visual quality than the base layer in the same area. The CSPS enhancement layer may reference the reconstructed pixels and motion vectors of the base layer corresponding to the same region.

[146] In embodiments, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers corresponding to each sub-image are contained in a separate track in the video file.

[147] In embodiments, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers with the same layer_id are contained in a separate track. In this example, the track corresponding to layer k includes only the CSPS layers corresponding to layer k .

[148] In embodiments, each CSPS layer of each sub-image is stored in a separate track. Each track may or may not have parsing or decoding dependencies on one or more other tracks.

[149] In embodiments, each track may contain bitstreams corresponding to layers i to j of the CSPS layers of all or a subset of sub-images, where 0<i=j=k, and k is the layer of the topmost CSPS layer.

[150] In embodiments, an image contains one or more types of associated media data, including a depth map, an alpha map, 3D geometry data, an occupancy map, etc. Such associated synchronized media data may be divided into one or multiple sub-data streams, each of which corresponds to one sub-image.

[151] In FIG. 11 shows an example of a video conference based on a method for creating multi-layer sub-images. The video stream contains one base layer video bitstream corresponding to a background image and one or more enhancement layer video bitstreams corresponding to foreground sub-images. Each video bitstream of the enhancement layer may correspond to a CSPS layer. The default display shows the image corresponding to the base layer. It contains one or more users' picture in picture (PIP). When a particular user is selected by the client control, the CSPS enhancement layer corresponding to the selected user may be decoded and displayed with improved quality or spatial resolution.

[152] In FIG. 12 is a flowchart illustrating an example of the above process. For example, in operation S1210, a video bitstream with multiple layers may be decoded. In operation S1220, a background and one or more foreground sub-images may be identified. In step S1230, it can be determined whether a specific sub-image area, such as one of the foreground sub-images, is selected. If a specific sub-image area is selected (YES in S1240), the enhanced sub-image can be decoded and displayed. If a specific sub-image area is not selected (NO in S1240), the background area can be decoded and displayed.

[153] In embodiments, a network intermediate unit (eg, a router) may select a subset of layers to send to the user based on its bandwidth. The image/sub-image organization can be used to adapt the bandwidth. For example, if the user does not have the bandwidth, the router splits the layers or selects some sub-images based on their importance or based on the settings used, and this can be done dynamically to adapt to the bandwidth.

[154] IN FIG. 13 shows an embodiment related to a 360-degree video use case. When a 360-degree spherical image, such as image 1310, is projected onto a planar image, the 360 projection image can be divided into a plurality of sub-images as a base layer. For example, the multiple sub-pictures may include an inverse stock picture, an upper stock image, a right stock image, a left stock image, a forward stock image, and a lower stock image. An enhancement layer of a particular sub-image, eg a direct sub-image, may be encoded and transmitted to the client. The decoder can decode both a base layer including all sub-images and an enhancement layer of a selected sub-image. When the current viewport is identical to the selected sub-image, the displayed image may have a higher quality with the decoded sub-image with an enhancement layer. Otherwise, the decoded image with the base layer may be displayed with lower quality.

[155] In embodiments, any layout information for display may be present in the file as additional information (eg, SEI messages or metadata). One or more decoded sub-pictures may be moved and displayed depending on the signaled arrangement information. The layout information may be signaled by the streaming server or broadcaster, or may be retrieved by the network entity or cloud server, or may be determined by the user's individual settings.

[156] In embodiments where the input image is divided into one or more (rectangular) sub-regions, each sub-region may be encoded as an independent layer. Each independent layer corresponding to a local area can have a unique layer_id value. For each independent layer, information about the size and location of the sub-image can be signaled. For example, image size (width, height), information about the offset of the upper left corner (x_offset, y_offset). FIG. 14 shows an example of a divided sub-image layout, size and position information of its sub-images, and its corresponding image prediction structure. Layout information, including sub-image size(s) and sub-image position(s), may be signaled in a high-level syntactic structure such as parameter set(s), a slice or tile group header, or an SEI message.

[157] In embodiments, each sub-image corresponding to an independent layer may have its own unique POC value within the AU. When a reference image among the images stored in the DPB is specified using syntax element(s) in an RPS or RPL structure, the POC value(s) of each sub-image corresponding to the layer may be used.

[158] In embodiments, to indicate the (inter-layer) prediction structure, the layer_id may not be used and a POC (delta) value may be used.

[159] In embodiments, a sub-image with a POC value equal to N corresponding to a layer (or local region) may or may not be used as a reference image of a sub-image with a POC value equal to N+K corresponding to the same layer (or the same local region). area) for motion compensated prediction. In most cases, the value of the number K can be equal to the maximum number of (independent) layers, which can be identical to the number of subregions.

[160] In the embodiments of FIGS. 15 shows an extended case of FIG. 14. When the input image is divided into multiple (eg four) sub-regions, each local region can be encoded with one or more layers. In this case, the number of independent layers may be equal to the number of sub-regions, and one or more layers may correspond to a sub-region. Thus, each sub-region may be encoded by one or more independent layers and zero or more dependent layers.

[161] In the embodiments of FIG. 15, the input image may be divided into four sub-regions. For example, the upper right subregion may be encoded as two layers, which are layer 1 and layer 4, while the lower right subregion may be encoded as two layers, which are layer 3 and layer 5. In this case, layer 4 may refer to layer 1 is for motion compensated prediction, while layer 5 can reference layer 3 for motion compensation.

[162] In embodiments, intra-loop filtering (eg, deblocking filtering, adaptive intra-loop filtering, shapeshifting, two-way filtering, or any deep learning-based filtering) across a layer boundary can be (optionally) disabled.

[163] In embodiments, motion compensated prediction or intra-block copying across a layer boundary can be (optionally) disabled.

[164] In embodiments, boundary padding for motion-compensated prediction or intra-loop filtering at the sub-image boundary may be processed optionally. A flag indicating whether boundary padding is processed or not may be signaled in a high-level syntactic structure, such as a parameter set(s) (VPS, SPS, PPS, or APS), a slice or tile group header, or an SEI message.

[165] In embodiments, the layout information of the sub-area(s) (or sub-image(s)) may be signaled to the VPS or SPS. Fig. 16A shows an example of syntax elements in a VPS, and FIG. 16B shows an example of syntax elements in the SPS. In this example, vps_sub_picture_dividing_flag is signaled to the VPS. The flag may indicate whether the input image(s) are divided into multiple subregions or not. When the value of vps_sub_picture_dividing_flag is 0, the input picture(s) in the encoded video sequence(s) corresponding to the current VPS cannot be divided into multiple sub-pictures. In this case, the size of the input image may be equal to the size of the encoded image (pic_width_in_luma_samples, pic_height_in_luma_samples) which is signaled in the SPS. When the value of vps_sub_picture_dividing_flag is 1, the input image(s) can be divided into multiple sub-pictures. In this case, the vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples syntax elements are signaled to the VPS. The values of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples can be equal to the width and height of the input image(s), respectively.

[166] In embodiments, the vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples values cannot be used for decoding, but can be used for layout and display.

[167] In embodiments, when the value of vps_sub_picture_dividing_flag is 1, the pic_offset_x and pic_offset_y syntax elements may be signaled in the SPS that correspond to (a) specific layer(s). In this case, the encoded image size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to a particular layer. Also, the position (pic_offset_x, pic_offset_y) of the upper left corner of the sub-region may be signaled to the SPS.

[168] In embodiments, the position information (pic_offset_x, pic_offset_y) of the upper left corner of the sub-region cannot be used for decoding, but can be used for layout and display.

[169] In embodiments, the layout information (size and position) of all or a sub-set of sub-regions of the input image(s), dependency information between layer(s) may be signaled in a parameter set or SEI message. FIG. 17 shows an example of syntax elements for specifying layout information of sub-regions, dependencies between layers, and a relationship between a sub-region and one or more layers. In this example, the num_sub_region syntax element specifies the number of (rectangular) subregions in the current encoded video sequence; The num_layers syntax element specifies the number of layers in the current encoded video sequence. The value of num_layers can be equal to or greater than the value of num_sub_region. When any subregion is encoded as a single layer, the value of num_layers can be equal to the value of num_sub_region. When one or more subregions are encoded as multiple layers, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] specifies the dependency from the j-th layer to the i-th layer; num_layers_for_region[i] specifies the number of layers associated with the i-th subregion; sub_region_layer_id[i][j] specifies the layer_id of the jth layer associated with the ith subregion. The values sub_region_offset_x [i] and sub_region_offset_y [i] indicate, respectively, the horizontal and vertical position of the upper left corner of the i-th subregion. The sub_region_width[i] and sub_region_height[i] values indicate the width and height of the i-th subregion, respectively.

[170] In embodiments, one or more syntax elements that define a set of output layers to indicate one or more layers to be output with or without profile tier level information may be signaled in a high-level syntax structure, such as VPS, DPS, SPS , PPS APS or SEI message. Referring to FIG. 18, a num_output_layer_sets syntax element indicating the number of output layer sets (OLS) in a coded video sequence related to the VPS may be signaled to the VPS. For each set of output layers, output_layer_flag can be signaled as many as there are output layers.

[171] In embodiments, output_layer_flag[i] equal to 1 may indicate that the i-th layer is output; vps_output_layer_flag[i] equal to 0 may indicate that the i-th layer is not output.

[172] In embodiments, one or more syntax elements that specify profile tier level information for each set of output layers may be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message. Still referring to FIG. 18, a num_profile_tile_level syntax element indicating the amount of profile tier level information on the OLS in the encoded video sequence related to the VPS may be signaled to the VPS. For each set of output layers, a set of syntax elements for profile tier level information, or an index indicating specific profile tier level information among entries in the profile tier level information, may be signaled as many as there are output layers.

[173] In embodiments, profile_tier_level_idx[i][j] may indicate an index in the profile_tier_level() syntax structure list in the VPS of the profile_tier_level() syntax structure that applies to the j-th layer of the i-th OLS.

[174] In embodiments, as shown in FIG. 20, the num_profile_tile_level and/or num_output_layer_sets syntax elements may be signaled when the number of maximum layers is greater than 1 (vps_max_layers_minus1 > 0).

[175] In embodiments, as shown in FIG. 20, a syntax element vps_output_layers_mode [i] indicating the output layer signaling mode for the i-th set of output layers may be present in the VPS.

[176] In embodiments, a vps_output_layers_mode[i] of 0 may indicate that only the topmost layer is output with the i-th set of output layers; vps_output_layer_mode[i] equal to 1 may indicate that all layers are output with the i-th set of output layers; vps_output_layer_mode[i] equal to 2 may indicate that the output layers are those with vps_output_layer_flag[i][j] equal to 1 with the i-th set of output layers. Other values may be reserved.

[177] In embodiments, output_layer_flag[i][j] may or may not be signaled depending on the value of vps_output_layers_mode[i] for the i-th set of output layers.

[178] In embodiments, with reference to FIGS. 19, the flag vps_ptl_signal_flag[i] may be present for the i-th set of output layers. Depending on the value of vps_ptl_signal_flag[i], the profile tier level information for the i-th set of output layers may or may not be signaled.

[179] In embodiments, as shown in FIG. 20, the number of subpics, max_subpics_minus1, in the current CVS may be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message.

[180] In embodiments, as shown in FIG. 21, a subpicture identifier, sub_pic_id[i], for the i-th subpicture may be signaled when the number of subpictures is greater than 1 (max_subpics_minus1 > 0).

[181] In embodiments, one or more syntax elements indicating a sub-image identifier belonging to each layer of each set of output layers may be signaled to the VPS. Referring to FIG. 20, sub_pic_id_layer[i][j][k], which indicates the k-th sub-image present in the j-th layer of the i-th set of output layers. With this information, the decoder can determine which sub-image can be decoded and output for each layer of a particular set of output layers.

[182] In an embodiment, a picture header (PH) is a syntax structure containing syntax elements that apply to all slices of an encoded picture. A picture unit (PU) is a set of NAL units that are related to each other according to a given classification rule, are sequential in decoding order, and contain exactly one encoded picture. The PU may contain a picture header (PH) and one or more NAL VCL units constituting the encoded picture.

[183] In an embodiment, the SPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId of 0, or provided through external means.

[184] In an embodiment, an SPS (RBSP) may be available to the decoding process before it is referenced and included in at least one AU with a TemporalId of 0 in a CVS that contains one or more PPSs. related to SPS, or provided through external means.

[185] In an embodiment, an SPS (RBSP) may be available to the decoding process before it is referenced by one or more PPSs, it is included in at least one PU with a nuh_layer_id equal to the smallest nuh_layer_id value of the NAL PPS units that refer to an SPS NAL unit in CVS that contains one or more SPS-related PPS or is provided through external means.

[186] In an embodiment, the SPS (RBSP) may be available to the decoding process before it is referenced by one or more PPSs, it is included in at least one PU with a TemporalId equal to 0 and nuh_layer_id equal to the smallest value nuh_layer_id of NAL PPS units that are assigned to a NAL SPS unit or provided through external means.

[187] In an embodiment, an SPS (RBSP) may be available to the decoding process before it is referenced by one or more PPSs, it is included in at least one PU with a TemporalId equal to 0 and nuh_layer_id equal to the smallest value of nuh_layer_id NAL PPS units that relate to a NAL SPS unit in a CVS that contains one or more PPS related to an SPS or is provided through external means.

[188] In the same or another embodiment, pps_seq_parameter_set_id may indicate the value of sps_seq_parameter_set_id for the referenced SPS. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in a coded layered video sequence (CLVS).

[189] In the same or another embodiment, all SPS NAL units with a particular sps_seq_parameter_set_id value in CVS may have the same content.

[190] In the same or another embodiment, regardless of the nuh_layer_id values, the SPS NAL units may share the same sps_seq_parameter_set_id value space.

[191] In the same or another embodiment, the nuh_layer_id value of the NAL SPS unit may be equal to the smallest nuh_layer_id value of the NAL PPS units that are related to the NAL SPS unit.

[192] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more PPS with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or ( direct or indirect) reference layer of a layer with nuh_layer_id equal to m.

[193] In an embodiment, the PPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the NAL PPS unit, or provided through external means.

[194] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced, it will be included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS that contains one or more PH (or coded slice NAL units) related to the PPS, or provided through external means.

[195] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it is included in at least one PU with nuh_layer_id equal to the smallest the nuh_layer_id value of the coded slice NAL units that relate to a PPS NAL unit in the CVS that contains one or more PHs (or coded slice NAL units) that relate to the PPS or are provided through external means.

[196] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units) included in at least one PU with a TemporalId equal to the TemporalId of the NAL unit PPS and nuh_layer_id equal to the smallest value of the nuh_layer_id of the coded slice NAL units that refer to the PPS NAL unit in the CVS that contains one or more PHs (or coded slice NAL units) that relate to the PPS, or are provided through external means.

[197] In the same or another embodiment, the ph_pic_parameter_set_id in the PH may indicate the pps_pic_parameter_set_id value for the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by encoded images in CLVS.

[198] In the same or another embodiment, all NAL PPS units with a particular pps_pic_parameter_set_id value in the PU may have the same content.

[199] In the same or another embodiment, regardless of nuh_layer_id values, NAL PPS units may share the same pps_pic_parameter_set_id value space.

[200] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the smallest nuh_layer_id value of the encoded slice NAL units that reference the PPS NAL unit.

[201] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more NAL units of an encoded slice with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n , or the (direct or indirect) reference layer of a layer with nuh_layer_id equal to m.

[202] In an embodiment, the PPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the NAL PPS unit, or provided through external means.

[203] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS that contains one or more PH (or coded slice NAL units) related to the PPS, or provided through external means.

[204] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it is included in at least one PU with nuh_layer_id equal to the smallest the nuh_layer_id value of the coded slice NAL units that relate to a PPS NAL unit in the CVS that contains one or more PHs (or coded slice NAL units) that relate to the PPS or are provided through external means.

[205] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units) included in at least one PU with a TemporalId equal to the TemporalId of the NAL unit PPS and nuh_layer_id equal to the smallest value of the nuh_layer_id of the coded slice NAL units that refer to the PPS NAL unit in the CVS that contains one or more PHs (or coded slice NAL units) that relate to the PPS, or are provided through external means.

[206] In the same or another embodiment, the ph_pic_parameter_set_id in the PH may indicate the pps_pic_parameter_set_id value for the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by encoded images in CLVS.

[207] In the same or another embodiment, all NAL PPS units with a particular pps_pic_parameter_set_id value in the PU may have the same content.

[208] In the same or another embodiment, regardless of nuh_layer_id values, NAL PPS units may share the same pps_pic_parameter_set_id value space.

[209] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the smallest nuh_layer_id value of the encoded slice NAL units that reference the PPS NAL unit.

[210] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more NAL units of an encoded slice with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n , or the (direct or indirect) reference layer of a layer with nuh_layer_id equal to m.

[211] When images are encoded into a bitstream that includes one or more layers with different resolutions, each encoded video sequence of the layer may use reference image resampling, where the reference image may have a different resolution than the current image. It is desirable to specify syntax elements that specify resampling of the reference image for spatial scalability or adaptive image resolution changes (ARC).

[212] As shown in FIG. 21, the output layer may indicate a layer of the output layer set that is output. An output layer set (OLS) may indicate a layer set containing a specified layer set, where one or more layers in the layer set are specified as output layers. The output layer set (OLS) layer index is the layer index in OLS to the list of layers in OLS.

[213] A sublayer may point to a temporary scalable layer of a temporary scalable bitstream including VCL NAL units with a particular TemporalId variable value and associated non-VCL NAL units. The sublayer representation may indicate a subset of the bitstream that includes the NAL units of a particular sublayer and lower sublayers.

[214] The RBSP VPS may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId of 0, or provided through external means. All VPS NAL units with a specific vps_video_parameter_set_id value in CVS can have the same content.

[215] As shown in FIG. 22, in embodiments, the vps_video_parameter_set_id provides a VPS identifier for reference by other syntax elements. The value of vps_video_parameter_set_id can be greater than 0.

[216] In embodiments, the value vps_max_layers_minus1 plus 1 may indicate the maximum allowed number of layers in each CVS referencing the VPS.

[217] The value vps_max_sublayers_ minus1 plus 1 may indicate the maximum number of temporary sublayers that can be present in a layer in each CVS referencing the VPS. The value of vps_max_sublayers_minus1 can be in the range from 0 to 6 inclusive.

[218] In embodiments, a vps_all_layers_same_num_sublayers_flag value of 1 may indicate that the number of temporary sublayers is the same for all layers in each CVS referencing the VPS. A vps_all_layers_same_num_sublayers_flag value of 0 may indicate that the layers in each CVS referencing the VPS may or may not have the same number of temporary sublayers. If absent, the value of vps_all_layers_same_num_sublayers_flag may be output to 1.

[219] In embodiments, a vps_all_independent_layers_flag equal to 1 may indicate that all layers in the CVS are independently encoded without using inter-layer prediction.

[220] In embodiments, a vps_all_independent_layers_flag equal to 0 may indicate that one or more layers in CVS can use inter-layer prediction. If absent, the value of vps_all_independent_layers_flag may be output to 1.

[221] In embodiments, vps_layer_id[i] may indicate the nuh_layer_id value of the i-th layer. For any two non-negative integer values m and n, when m is less than n, the value of vps_layer_id[m] can be less than vps_layer_id[n].

[222] In embodiments, a vps_independent_layer_flag[i] value of 1 may indicate that layer index i does not use inter-layer prediction. A vps_independent_layer_flag[i] value of 0 may indicate that layer index i can use cross-layer prediction, and vps_direct_ref_layer_flag[i][j] syntax elements for j in the range 0 to i - 1 inclusive are present in the VPS. If absent, the value of vps_independent_layer_flag [i] may be output to 1.

[223] In embodiments, a vps_direct_ref_layer_flag[i][j] value of 0 may indicate that layer index j is not a direct reference layer for layer index i. A vps_direct_ref_layer_flag[i][j] value of 1 may indicate that layer index j is the direct reference layer for layer index i. When vps_direct_ref_layer_flag[i][j] is missing for i and j in the range 0 to vps_max_layers_minus1 inclusive, it may be output equal to 0. When vps_independent_layer_flag[i] is 0, there may be at least one value of j in the range 0 to i - 1 inclusive, so the value of vps_direct_ref_layer_flag[i][j] is 1.

[224] The variables NumDirectRefLayers[i], DirectRefLayerIdx[i][d], NumRefLayers[i], RefLayerIdx[i][r], and LayerUsedAsRefLayerFlag[j] can be inferred as follows:

[225] The variable GeneralLayerIdx[i], which specifies the layer index for a layer with nuh_layer_id equal to vps_layer_id[i], can be obtained as follows:

[226] For any two different values of i and j, both in the range 0 to vps_max_layers_minus1 inclusive, when dependencyFlag[i][j] is 1, there may be a bitstream matching requirement so that the values of chroma_format_idc and bit_depth_minus8 that apply to i- th layer could be equal to the values of chroma_format_idc and bit_depth_minus8, respectively, which are applied to the j-th layer.

[227] A max_tid_ref_present_flag[i] value of 1 may indicate that the max_tid_il_ref_pics_plus1[i] syntax element is present. A max_tid_ref_present_flag[i] value of 0 may indicate that the max_tid_il_ref_pics_plus1[i] syntax element is missing.

[228] A max_tid_il_ref_pics_plus1[i] value of 0 may indicate that inter-layer prediction is not used by non-IRAP images of the i-th layer. A value of max_tid_il_ref_pics_plus1 [i] greater than 0 may indicate that for decoding the i-th layer images, no image with a TemporalId greater than max_tid_il_ref_pics_plus1 [i] - 1 is used as an ILRP. If absent, the value max_tid_il_ref_pics_plus1 [i] can be output as 7.

[229] A value of each_layer_is_an_ols_flag equal to 1 may indicate that each OLS contains only one layer, and each layer itself in the CVS related to the VPS is an OLS with a single layer included, which is the only output layer. An each_layer_is_an_ols_flag value of 0 means that an OLS can contain more than one layer. If vps_max_layers_minus1 is 0, the value of each_layer_is_an_ols_flag can be output as 1. Otherwise, when vps_all_independent_layers_flag is 0, the value of each_layer_is_an_ols_flag can be output as 0.

[230] An ols_mode_idc value of 0 may indicate that the total number of OLSs specified by the VPS is vps_max_layers_minus1 + 1, where the i-th OLS includes layers with layer indices from 0 to i inclusive, and for each OLS only the layer is output the highest level in OLS.

[231] An ols_mode_idc value of 1 may indicate that the total number of OLSs specified by the VPS is vps_max_layers_minus1 + 1, where the i-th OLS includes layers with layer indices from 0 to i inclusive, and for each OLS all layers are output in OLS.

[232] An ols_mode_idc value of 2 may indicate that the total number of OLSs indicated by the VPS are explicitly signaled, and for each OLS, the output layers are explicitly signaled and the other layers are layers that are direct or indirect reference layers of the OLS output layers.

[233] The value of ols_mode_idc can range from 0 to 2, inclusive. Value 3 ols_mode_idc reserved for future use by ITU-T | ISO/IEC.

[234] When vps_all_independent_layers_flag is 1 and each_layer_is_an_ols_flag is 0, the value of ols_mode_idc can be output to 2.

[235] num_output_layer_sets_minus1 plus 1 can indicate the total number of OLS specified by the VPS when ols_mode_idc is 2.

[236] The TotalNumOlss variable indicating the total number of OLSs specified by the VPS can be output as follows:

[237] ols_output_layer_flag[i][j] equal to 1 can indicate that the layer with nuh_layer_id equal to vps_layer_id[j] is the output layer of the i-th OLS when ols_mode_idc is equal to 2. ols_output_layer_flag[i][j] is equal to 0 can indicate that the layer with nuh_layer_id equal to vps_layer_id[j] is not the output layer of the i-th OLS when ols_mode_idc is equal to 2.

[238] Variable NumOutputLayersInOls [i], indicating the number of output layers in the i-th OLS, variable NumSubLayersInLayerInOLS [i] [j], indicating the number of sublayers in the j-th layer in the i-th OLS, variable OutputLayerIdInOls [i] [j] , which determines the nuh_layer_id value of the j-th output layer in the i-th OLS, and the variable LayerUsedAsOutputLayerFlag [k], which determines whether the k-th layer is used as an output layer in at least one OLS, can be obtained as follows:

[239] For each value of i in the range 0 to vps_max_layers_minus1 inclusive, the values LayerUsedAsRefLayerFlag[i] and LayerUsedAsOutputLayerFlag[i] cannot be equal to 0 at the same time. In other words, there may be no layer that is neither an output layer of at least one OLS nor a direct reference layer of any other layer.

[240] For each OLS there may be at least one layer that is an output layer. In other words, for any value of i in the range 0 to TotalNumOlss - 1 inclusive, the value of NumOutputLayersInOls[i] can be greater than or equal to 1.

[241] You can get the NumLayersInOls [i] variable that defines the number of layers in the i-th OLS, and the LayerIdInOls [i] [j] variable that defines the nuh_layer_id value of the j-th layer in the i-th OLS as follows:

[242] The OlsLayerIdx[i][j] variable defining the OLS layer index for a layer with nuh_layer_id equal to LayerIdInOls[i][j] is obtained as follows:

[243] The lowest layer in each OLS may be an independent layer. In other words, for every i in the range 0 to TotalNumOlss - 1 inclusive, the value of vps_independent_layer_flag [GeneralLayerIdx [LayerIdInOls [i] [0]]] can be equal to 1.

[244] Each layer can be included in at least one OLS specified by the VPS. In other words, for each layer with a particular nuh_layer_id nuhLayerId value equal to one of vps_layer_id[k] for k in the range 0 to vps_max_layers_minus1 inclusive, there can be at least one pair of values i and j, where i is in the range 0 to TotalNumOlss - 1 inclusive, and j is in the range NumLayersInOls[i] - 1 inclusive, so the value of LayerIdInOls[i][j] is nuhLayerId.

[245] In embodiments, for the current CurrPic picture, the decoding process may be performed as follows:

- PictureOutputFlag can be set as follows:

- If one of the following conditions is true, PictureOutputFlag can be set to 0:

- the current image is a RASL image, and the NoOutputBeforeRecoveryFlag value of the associated IRAP image is 1.

- gdr_enabled_flag is equal to 1 and the current image is a GDR image with NoOutputBeforeRecoveryFlag equal to 1.

- the gdr_enabled_flag syntax element is equal to 1, the current image is associated with a GDR image with NoOutputBeforeRecoveryFlag equal to 1, and the PicOrderCntVal of the current image is less than the RpPicOrderCntVal of the associated GDR image;

- sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0, and the current AU contains an image picA that satisfies all of the following conditions:

- PicA has a PictureOutputFlag equal to 1.

- PicA nuh_layer_id has a nuhLid greater than that of the current image.

- PicA belongs to the OLS output layer (i.e. OutputLayerIdInOls[TargetOlsIdx][0] is equal to nuhLid).

- sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 2, and ols_output_layer_flag [TargetOlsIdx] [GeneralLayerIdx [nuh_layer_id]] is equal to 0.

- Otherwise, PictureOutputFlag may be set to pic_output_flag.

[246] After all slices of the current image have been decoded, the current decoded image is marked as "used for short-term reference", and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as "used for short-term reference".

[247] In embodiments where each layer is a set of output layers, PictureOutputFlag may be set to pic_output_flag regardless of the value of ols_mode_idc.

[248] In embodiments, PictureOutputFlag is set to 0 when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is 0, ols_mode_idc is 0, and the current AU contains an image picA that satisfies all of the following conditions: PicA has a PictureOutputFlag equal to 1, PicA has nuh_layer_id nuhLid greater than the current image, and PicA belongs to the OLS output layer (that is, OutputLayerIdInOls[TargetOlsIdx][0] equals nuhLid).

[249] In embodiments, PictureOutputFlag may be set to 0 when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is 0, ols_mode_idc is 2, and ols_output_layer_flag [TargetOlsIdx] [GeneralLayerIdx] [GeneralLayerIdx] is 0.

[250] Resampling of a reference image can allow adaptive resampling of the encoded (multi-layer) video sequence and spatial scalability across layers that have dependencies between layers belonging to the same set of output layers.

[251] In the embodiments shown in FIGS. 23, sps_ref_pic_resampling_enabled_flag is signaled in a parameter set (eg, a sequence parameter set). The sps_ref_pic_resampling_enabled_flag may indicate whether reference picture resampling is used to adaptively resample the encoded video sequence referencing the SPS or for spatial scaling across layers. An sps_ref_pic_resampling_enabled_flag equal to 1 may indicate that reference picture resampling is enabled and one or more image slices in the CLVS refer to a reference picture with a different spatial resolution in the active reference picture list entry. An sps_ref_pic_resampling_enabled_flag of 0 may indicate that reference picture resampling is disabled and that no image slice in CLVS references a reference picture of a different spatial resolution in the active reference picture list entry.

[252] In embodiments, when sps_ref_pic_resampling_enabled_flag is 1, for the current image, a reference image with a different spatial resolution either belongs to the same layer or a different layer than the layer containing the current image.

[253] In another embodiment, sps_ref_pic_resampling_enabled_flag equal to 1 may indicate that resampling of the reference picture is enabled, and one or more image slices in the CLVS reference a reference picture with a different spatial resolution or a different scaling window in the active reference picture list entry. However, an sps_ref_pic_resampling_enabled_flag of 0 may indicate that reference picture resampling is disabled and that no image slice in CLVS references a reference picture with a different spatial resolution or a different scaling window in the active reference picture list entry.

[254] In embodiments, when sps_ref_pic_resampling_enabled_flag is equal to 1, for the current image, a reference image with a different spatial resolution or a different zoom window either belongs to the same layer or a different layer than the layer containing the current image.

[255] In embodiments, sps_res_change_in_clvs_allowed_flag may indicate whether the image resolution is changed within CLVS or CVS or not. However, sps_res_change_in_clvs_allowed_flag equal to 1 may indicate that the spatial resolution of the image can change within the CLVS with reference to the SPS. However, an sps_res_change_in_clvs_allowed_flag of 0 may indicate that the spatial resolution of the image is not changed within any CLVS referencing the SPS. If missing, sps_res_change_in_clvs_allowed_flag may be output as 0.

[256] In embodiments where sps_ref_pic_resampling_enabled_flag is 1 and sps_res_change_in_clvs_allowed_flag is 0, reference image resampling can only be used for spatial scalability and not for adaptive resolution resampling in CLVS.

[257] In embodiments where sps_ref_pic_resampling_enabled_flag is equal to 1 and sps_res_change_in_clvs_allowed_flag is equal to 1, reference image resampling can be used for both spatial scalability and adaptive resolution resampling in CLVS.

[258] When sps_ref_pic_resampling_enabled_flag is 1, sps_res_change_in_clvs_allowed_flag is 0, and sps_video_parameter_set_id is 0, pps_scaling_window_explicit_signalling_flag can be 1. This implies that the scaling window parameters must be explicitly signaled, rather than inferring values from the consistency window parameters when the image resolution is constant in within CLVS or CVS and oversampling of the reference image is used.

[259] In embodiments, sps_virtual_boundaries_present_flag is signaled to the SPS, as shown in FIG. 23. The sps_virtual_boundaries_present_flag may indicate whether virtual boundary information is signaled to the SPS or not.

[260] In embodiments, sps_virtual_boundaries_present_flag is signaled under certain circumstances only when sps_res_change_in_clvs_allowed_flag is 0 because when reference picture resampling is used, virtual boundary information may not be signaled in the SPS.

[261] In embodiments, sps_virtual_boundaries_present_flag equal to 1 may indicate that virtual boundary information is signaled to the SPS. However, sps_virtual_boundaries_present_flag equal to 0 may indicate that virtual boundary information is not signaled to the SPS. When one or more virtual boundaries are signaled in the SPS, intra-loop filtering operations across the virtual boundaries are disabled on images referencing the SPS. Intra-loop filtering operations include deblocking filter, adaptive sample offset filter, and adaptive loop filter operations. If absent, sps_virtual_boundaries_present_flag is output as 0.

[262] In embodiments, sps_subpic_info_present_flag is signaled to the SPS, as shown in FIG. 23. The sps_subpic_info_present_flag may indicate whether the subpicture division information is signaled in the SPS or not.

[263] In embodiments, sps_subpic_info_present_flag is conditionally signaled only when sps_res_change_in_clvs_allowed_flag is 0 because when reference picture resampling is used, subpicture division information may not be signaled in the SPS.

[264] In embodiments, an sps_subpic_info_present_flag equal to 1 may indicate that subframe information is present for the CLVS, and there may be one or more subframes in each CLVS picture. sps_subpic_info_present_flag equal to 0 may indicate that there is no subpicture information for the CLVS and that there is only one subpicture in each CLVS picture. If missing, sps_subpic_info_present_flag may be output as 0.

[265] In embodiments, pps_res_change_in_clvs_allowed_flag may be signaled in the PPS, as shown in FIG. 24. The value of pps_res_change_in_clvs_allowed_flag in the PPS may be equal to the value of sps_res_change_in_clvs_allowed_flag in the SPS referenced by the PPS.

[266] In the same embodiment, image width and height information can be passed to the PPS only when the value of pps_res_change_in_clvs_allowed_flag is 1. When pps_res_change_in_clvs_allowed_flag is 0, the image width and height values are inferred to be equal to the maximum image width and height values passed to the SPS .

[267] In embodiments, pps_pic_width_in_luma_samples may indicate the width of each decoded picture with reference to the PPS in units of luma samples. In this case, pps_pic_width_in_luma_samples must not be equal to 0, must be an integer multiple of Max (8, MinCbSizeY), and must be less than or equal to sps_pic_width_max_in_luma_samples. If missing, the value of pps_pic_width_in_luma_samples may be output to be equal to sps_pic_width_max_in_luma_samples. When sps_ref_wraparound_enabled_flag is 1, the value (CtbSizeY / MinCbSizeY + 1) must be less than or equal to the value (pps_pic_width_in_luma_samples / MinCbSizeY - 1). In this case, pps_pic_height_in_luma_samples can indicate the height of each decoded image with reference to PPS in units of luminance samples. In this case, pps_pic_height_in_luma_samples must not be equal to 0, must be an integer multiple of Max(8, MinCbSizeY), and must be less than or equal to sps_pic_height_max_in_luma_samples. If missing, the value of pps_pic_height_in_luma_samples may be inferred to be equal to sps_pic_height_max_in_luma_samples.

[268] In FIG. 25A-25C are block diagrams illustrating exemplary processes 2500A, 2500B, and 2500C for decoding an encoded video bitstream. In some implementations, one or more process blocks of FIG. 25A-25C may be executed by decoder 210. In some implementations, one or more process blocks of FIGS. 25A-25C may be performed by another device or group of devices separate from or including decoder 210, such as encoder 203.

[269] As shown in FIG. 25A, process 2500A may include obtaining from the coded video bitstream a first flag indicating that resampling of a reference picture is enabled for the coded video sequence (CVS) (block 2511). In embodiments, the first flag may correspond to sps_ref_pic_resampling_enabled_flag.

[270] As further shown in FIG. 25A, process 2500A may include determining the value of a second flag indicating whether the image resolution is changed in CVS (block 2512). In embodiments, the first flag may correspond to sps_res_change_in_clvs_allowed_flag.

[271] As further shown in FIG. 25A, process 2500A may include determining whether the value of the second flag indicates whether the image resolution is changed in CVS (block 2513).

[272] As further shown in FIG. 25A, process 2500A may include, based on the value of a second flag indicating that the image resolution is being changed in CVS (YES at block 2515), decoding the CVS using reference image resampling for spatial scalability and adaptive resolution resampling (ARC) (block 2514 ).

[273] As further shown in FIG. 25A, process 2500A may include, based on the value of the second flag indicating that the image resolution is not changed in CVS (NO at block 2513), decoding the CVS using reference picture resampling for spatial scalability and without using reference picture resampling for ARC ( block 2515).

[274] As shown in FIG. 25B, process 2500B may include determining whether the value of the second flag indicates whether the image resolution is changed in CVS (block 2523). In embodiments, block 2523 of process 2500B may correspond to block 2513 of process 2500A.

[275] As further shown in FIG. 25B, process 2500B may include, based on the value of a second flag indicating that the image resolution is not changed in CVS (NO at block 2523), obtaining a third flag indicating whether virtual boundary information is signaled (block 2525). In embodiments, the third flag may correspond to sps_virtual_boundaries_present_flag.

[276] As further shown in FIG. 25B, process 2500B may include: based on the value of a second flag indicating that the image resolution is being changed in CVS (YES at block 2523), the third flag may not be signaled (block 2524).

[277] As shown in FIG. 25C, process 2500C may include determining whether the value of the second flag indicates whether the image resolution is changed in CVS (block 2533). In embodiments, block 2533 of process 2500C may correspond to block 2513 of process 2500A.

[278] As further shown in FIG. 25C, process 2500C may include, based on the value of a second flag indicating that the image resolution is not changed in CVS (NO at block 2533), obtaining a fourth flag indicating whether sub-image division information is signaled (block 2535). In embodiments, the fourth flag may correspond to sps_subpic_info_present_flag.

[279] As further shown in FIG. 25C, process 2500C may include: based on the value of a second flag indicating that the image resolution is being changed in CVS (YES at block 2533), the fourth flag may not be signaled (block 2534).

[280] In embodiments, one or more process blocks 2500A, 2500B, and 2500C may be combined in any order. In addition, although in FIG. 25A-25C show exemplary process blocks 2500A, 2500B, and 2500C; in some implementations, processes 2500A, 2500B, and 2500C may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those depicted in FIG. 25A-25C. Additionally or alternatively, two or more process blocks 2500A, 2500B, and 2500C may be executed in parallel.

[281] In addition, the proposed methods may be implemented by processing circuits (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program that is stored on a non-transitory computer-readable medium to implement one or more of the proposed methods.

[282] The methods described above may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 26 illustrates a computer system 2600 suitable for implementing certain embodiments of the disclosed subject matter.

[283] Computer software may be encoded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed directly or through interpretation, execution of microcode, and etc., computer central processing units (CPU), graphic processing units (GPU), etc.

[284] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[285] The components shown in FIG. 26 for computer system 2600 are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of computer software implementing embodiments of the present invention. The configuration of components should also not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system 2600.

[286] Computer system 2600 may include certain human-machine interface input devices. Such a human-machine interface input device may respond to input from one or more human users through, for example, tactile input (e.g., key presses, swipes, control glove movement), audio input (e.g., voice, hand clapping), visual input (for example: gestures), olfactory input (not shown). Human-machine interface devices can also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, ambient sound), images (e.g. scanned images, photographic images captured from a camera for still images), video (e.g. 2D video, 3D video, including stereoscopic video).

[287] Human-machine interface input devices may include one or more of (only one of each illustrated): keyboard 2601, mouse 2602, trackpad 2603, touch screen 2610 and associated graphics adapter 2650, control glove, joystick 2605, microphone 2606, scanner 2607, camera 2608.

[288] The computer system 2600 may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such HMI output devices may include haptic output devices (e.g., haptic feedback from touch screen 2610, control glove, or joystick 2605, but there may also be haptic feedback devices that do not serve as input devices, devices audio output devices (for example: speakers 2609, headphones (not shown)), image output devices (for example, screens 2610, including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have input capabilities touch screens, each with or without haptic feedback capability - some of which may be capable of producing two-dimensional visual output or more than three-dimensional output through means such as stereographic output; virtual reality glasses (not shown), holographic displays and smoke displays tanks (not shown)) and printers (not shown).

[289] The computer system (2600) may also include human accessible storage devices and associated media, such as optical media, including CD/DVD ROM/RW (2620) with CD/DVD or similar media (2621), flash drive (2622), removable hard drive or solid state drive (2623), previously developed magnetic media such as tape and floppy disk (not pictured), specialized ROM/ASIC/PLD based devices such as dongles (not pictured) and so on.

[290] Those skilled in the art will also understand that the term "machine readable medium" as used in connection with the subject matter disclosed herein does not cover transmission media, carrier waves, or other time signals.

[291] The computer system (2600) may also include an interface to one or more communications networks (955). Networks can be, for example, wireless, wired, optical. Additionally, networks can be LAN, WAN, metropolitan, automotive and industrial, real-time, latency tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless local area networks (LAN), cellular networks, including global systems for mobile communications (GSM), third generation (3G), fourth generation (4G), fifth generation (5G), Long Term Evolution (LTE) networks, etc., wired television or wireless global digital networks, including cable television, satellite television and terrestrial broadcast television, automotive and industrial, including CAN bus, and so on. Some networks typically require external network interface adapters (954) that are connected to certain general purpose data ports or peripheral buses (949) (for example, the universal serial bus (USB) ports of a 2600 computer system); others are typically integrated into the core of the computer system 2600 by connecting to a system bus, as described below (for example, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). For example, network 2655 may be connected to peripheral bus 2649 using network interface 2654. Using any of these networks, computer system 2600 can communicate with other entities. Such communication may be unidirectional, receive only (for example, broadcast television), unidirectional, send only (for example, CAN bus to certain CAN bus devices), or bidirectional, such as for other computer systems using a local or wide area digital network . Each of these networks and network interfaces (954) may use specific protocols and protocol stacks, as described above.

[292] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core 2640 of the computer system 2600.

[293] The core (2640) may include one or more central processing units (CPUs) (2641), graphics processing units (GPUs) (2642), field programmable gate area (FPGA) processors (2643), hardware accelerators ( 2644) for specific tasks, etc. These devices, along with read-only memory (ROM) 2645, random access memory (RAM) 2646, internal mass storage such as internal hard drives not accessible to the user, solid-state drives (SSD ) etc. 2647, can be connected via the 2648 system bus. On some computer systems, the 2648 system bus may be available as one or more physical connectors to allow expansion with additional CPUs, GPUs, etc. Peripherals can be connected either directly to the core system bus 2648 or via a peripheral bus 2649. The peripheral bus architecture includes peripheral component interconnection (PCI), USB, etc.

[294] The CPU 2641, GPU 2642, FGPA 2643, and accelerators 2644 may execute certain instructions that, in combination, may constitute the aforementioned computer code. This computer code may be stored in ROM 2645 or RAM 2646. Transient data may also be stored in RAM 2646, while persistent data may be stored, for example, in the internal mass storage device 2647. Rapid storage and retrieval to either storage device can be accessed by using a cache memory that may be closely coupled to one or more of the CPU 2641, GPU 2642, mass storage device 2647, ROM 2645, RAM 2646, etc.

[295] A computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specifically designed and constructed for the purposes of the present invention, or they may be well known and available to those skilled in the computer software art.

[296] By way of example, and not limitation, a computer system having an architecture (2600) and, in particular, a core (2640), may provide functionality through processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.). etc.) executing software embodied on one or more tangible machine-readable media. Such computer-readable media may be media associated with user-accessible mass storage, as described above, as well as certain storage devices of core 2640 that are non-transitory in nature, such as internal mass storage 2647 or ROM 2645. Software implementing various embodiments implementation of the present invention may be stored in such devices and executed by the core 2640. The computer readable medium may include one or more storage devices or chips according to specific needs. Software may cause the core (2640) and, in particular, processors therein (including CPUs, GPUs, FPGAs, etc.) to execute certain processes or certain portions of specific processes described herein, including defining data structures stored in RAM (2646) and modifying such data structures according to processes defined by the software. In addition or alternatively, a computer system may provide functionality as a result of logic built into hardware circuitry or otherwise embodied in circuitry (e.g., accelerator 2644), which may operate instead of or in conjunction with software to perform certain processes or portions of specific processes described in this document. A reference to software may include logic and vice versa where applicable. Reference to a computer-readable medium may include circuitry (eg, an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both, where applicable. The present invention includes any suitable combination of hardware and software.

[297] Although this description discloses several exemplary embodiments, there are variations, permutations, and various substitute equivalents that fall within the scope of the invention. Thus, it will be appreciated that those skilled in the art will be able to develop numerous systems and methods that, although not expressly shown or described herein, embody the principles of the invention and are thus within the spirit and scope thereof.

Claims (42)

1. A method for decoding an encoded video bitstream using at least one processor, comprising: obtaining from the coded video bitstream a first flag indicating that resampling of the reference picture is enabled for the coded video sequence (CVS); determining a value of a second flag indicating whether the image resolution is changed in CVS; based on the value of the second flag indicating that the image resolution is changed in the CVS, decoding the CVS using resampling of the reference image for spatial scalability and adaptive resolution resampling (ARC); And based on the value of the second flag indicating that the image resolution is not changed in CVS, CVS decoding using reference picture resampling for spatial scalability and without using reference picture resampling for ARC, wherein the first flag and the second flag are signaled in a sequence parameter set (SPS); And wherein, based on the value of the second flag indicating that the image resolution is not changed in CVS, a fifth flag included in the Picture Parameter Set (PPS) referring to the SPS indicates that the image resolution is not changed in CVS, wherein, based on the fifth flag indicating that the image resolution is not changed in the CVS, at least one of the width and height of the image corresponding to the PPS is not explicitly signaled, and wherein at least one of the width and the height is output equal to at least one of the maximum width and maximum height signaled in the SPS. 2. The method of claim 1, wherein the CVS is a coded layered video sequence (CLVS). 3. The method of claim 1, wherein the second flag is not signaled in the encoded video bitstream, and the value of the second flag is output to indicate that the image resolution is not changed in CVS. 4. The method according to claim 1, additionally containing: based on the value of the second flag indicating that the image resolution is not changed in CVS, obtaining a third flag indicating whether virtual boundary information is signaled. 5. The method according to claim 1, additionally containing: based on the value of the second flag indicating that the image resolution is not changed in CVS, obtaining a fourth flag indicating whether sub-image division information is signaled. 6. The method of claim 1, wherein based on the value of the second flag indicating that the image resolution is not changed in CVS, no zoom window parameters are signaled. 7. The method of claim 1, wherein, based on the value of the second flag indicating that the image resolution is changed in CVS, the fifth flag included in the picture parameter set (PPS) referring to the SPS indicates that the image resolution is changed in CVS, And wherein, based on the fifth flag indicating that the image resolution is changed in the CVS, at least one of the width and height of the image corresponding to the PPS is explicitly signaled. 8. A device for decoding an encoded video bitstream, comprising: at least one memory element configured to store program code; And at least one processor configured to read the program code and operate in accordance with the instructions of the program code, the program code including: a first acquisition code configured to cause the at least one processor to obtain, from the coded video bitstream, a first flag indicating that reference picture resampling is enabled for the coded video sequence (CVS); determination code configured to cause the at least one processor to determine the value of a second flag indicating whether the image resolution is changed in CVS; a first decoding code configured to cause the at least one processor, based on the value of the second flag indicating that the image resolution is changed in the CVS, to decode the CVS using reference image resampling for spatial scalability and adaptive resolution resampling (ARC); And a second decoding code configured to cause the at least one processor, based on the value of the second flag indicating that the image resolution is not changed in the CVS, to decode the CVS using reference picture resampling for spatial scalability and without using reference picture resampling for ARC; wherein the first flag and the second flag are signaled in the Sequence Parameter Set (SPS), wherein, based on the value of the second flag indicating that the image resolution is not changed in CVS, a fifth flag included in the Picture Parameter Set (PPS) referring to the SPS indicates that the image resolution is not changed in CVS, wherein, based on the fifth flag indicating that the image resolution is not changed in the CVS, at least one of the width and height of the image corresponding to the PPS is not explicitly signaled, and wherein at least one of the width and the height is output equal to at least one of the maximum width and maximum height signaled in the SPS. 9. The apparatus of claim 8, wherein the CVS is a coded layered video sequence (CLVS). 10. The apparatus of claim 8, wherein the second flag is not signaled in the encoded video bitstream, and the value of the second flag is output to indicate that the image resolution is not changed in CVS. 11. The device according to claim 8, in which the program code additionally includes: a second acquisition code configured to cause the at least one processor, based on the value of the second flag indicating that the image resolution is not changed in the CVS, to obtain a third flag indicating whether the virtual boundary information is signaled. 12. The device according to claim 8, in which the program code additionally includes: a third acquisition code configured to cause the at least one processor, based on the value of the second flag indicating that the image resolution is not changed in the CVS, to obtain a fourth flag indicating whether sub-image division information is signaled. 13. The apparatus of claim 8, wherein based on the value of the second flag indicating that the image resolution is not changed in CVS, no zoom window parameters are signaled. 14. The apparatus of claim 8, wherein, based on the value of the second flag indicating that the image resolution is changed in CVS, the fifth flag included in the picture parameter set (PPS) referring to the SPS indicates that the image resolution is changed in CVS, And wherein, based on the fifth flag indicating that the image resolution is changed in the CVS, at least one of the width and height of the image corresponding to the PPS is explicitly signaled. 15. A non-transitory computer-readable medium storing instructions containing: one or more instructions that, when executed by one or more processors of a video bitstream decoding device, instruct one or more processors to: carry out the steps of the method according to any one of paragraphs. 1-7.

Images