RU2814858C1 - Method of signaling division of rectangular slice in encoded video stream - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to video encoding means. Video data containing one or more sub-images are received. Number of sub-images and the delta value between the number of sub-images and the number of rectangular slices are signaled. Number of rectangular slices is obtained based on the number of sub-images and the delta value. When the current image is not the first image of a coded layer video sequence (CLVS), for each value i in range from 0 to the syntax element sps_num_subpics_minus1, if the value of the variable SubpicIdVal [ i ] of the current image is not equal to the value of SubpicIdVal [ i ] of the previous image in the decoding order in the same layer, syntax element subpic_treated_as_pic_flag[ i ] and syntax element loop_filter_across_subpic_enabled_flag[ i ] are equal to 1, wherein the syntax element sps_num_subpics_minus1 plus 1 indicates the number of sub-images in each image in the CLVS.

EFFECT: high efficiency of video encoding.

20 cl, 26 dwg

Description

CROSS REFERENCE TO RELATED APPLICATION

[1] This application claims the benefit of U.S. Provisional Patent Application No. 63/003,101, filed March 31, 2020, and U.S. Patent Application No. 17/098,892, filed November 16, 2020, in the United States Patent and Trademark Office, which are incorporated herein by reference in their entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[2] The disclosed subject matter of the invention relates to video encoding and decoding, and more particularly to slice division signaling in an encoded video stream.

BACKGROUND OF THE ART

[3] Video encoding and decoding using motion-compensated inter-frame prediction has been known for decades. Uncompressed digital video may consist of a series of images, each image having a spatial dimension, such as 1920 x 1080 luminance samples and associated chrominance samples. A series of images can have a fixed or variable image rate (informally also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 luminance samples resolution at 60 Hz frame rate) requires a bandwidth of close to 1.5 Gbps. An hour of such video requires more than 600 GB of disk space.

[4] One of the purposes of video encoding and decoding may be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, or a combination of both, can be used. Lossless compression refers to techniques in which an exact copy of the original signal can be reconstructed from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal suitable for the intended application. In the case of video, lossy compression is widely used. The acceptable degree of distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of broadcast television applications. The achievable compression ratio may reflect the following: Higher resolution/acceptable distortion may result in a higher compression ratio.

[5] The video encoder and video decoder may use techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding, some of which will be presented below.

[6] Historically, video encoders and video decoders have tended to operate at a given picture size, which in most cases was defined and remained constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2 it is known that the system design changes the horizontal resolution (and thus the image size) depending on factors such as scene activity, but only for I-pictures, hence typically for GOP. Resampling of reference images to use different resolutions in CVS is known, for example, from ITU-T Rec. H.263, Annex P. However, here the image size is not changed, only the reference images are resampled, resulting in potentially only parts of the image canvas being used (in in the case of downsampling) or capturing only parts of the scene (in the case of upsampling). In addition, H.263 Annex Q allows an individual macroblock to be resampled by a factor of two (in each dimension), up or down. Again, the image size remains the same. The macroblock size is fixed in H.263 and therefore does not need to be signalled.

[7] Image resizing in predicted images has become more common in modern video coding. For example, VP9 allows you to resample a reference image and change the resolution of the entire image. Likewise, some proposals made regarding VVC (including, for example, Hendry, et. al, “On adaptive resolution change (ARC) for VVC,” JVT document JVET-M0135-v1, January 9-19 2019, incorporated herein in its entirety) allow entire reference images to be resampled to a different, higher or lower, resolution. This document proposes to encode different resolution candidates in a sequence parameter set and refer to them using syntax elements for each image in the image parameter set.

DISCLOSURE OF THE INVENTION

[8] Embodiments of the invention relate to a method, system, and computer-readable medium for encoding video data. In one aspect, a method for encoding video data is provided. The method may include receiving video data including one or more sub-images. The number of sub-images and the delta value between the number of sub-images and the number of rectangular slices are signaled. The number of rectangular slices is obtained based on the number of sub-images and the delta value.

[9] In another aspect, a computer system for encoding video data is provided. The computer system may include one or more processors, one or more computer-readable memory elements, one or more computer-readable physical storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of one or more more processors through at least one of one or more memory elements, whereby the computer system is capable of implementing the method. The method may include receiving video data including one or more sub-images. The number of sub-images and the delta value between the number of sub-images and the number of rectangular slices are signaled. The number of rectangular slices is obtained based on the number of sub-images and the delta value.

[10] In another aspect, a computer-readable medium for encoding video data is provided. A computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of one or more tangible storage devices, the program instructions being executed by a processor. The program instructions are executed by the processor to implement the method, which accordingly may include receiving video data including one or more sub-images. The number of sub-images and the delta value between the number of sub-images and the number of rectangular slices are signaled. The number of rectangular slices is obtained based on the number of sub-images and the delta value.

BRIEF DESCRIPTION OF THE DRAWINGS

[11] These and other objects, features and advantages will become apparent from the following detailed description of the illustrative embodiments, which should be read in conjunction with the accompanying drawings. Various elements of the drawings are not to scale because the illustrations are provided for clarity and ease of understanding by one skilled in the art in conjunction with the detailed description. On the drawings:

[12] FIG. 1 is a schematic illustration of a simplified block diagram of a communication system according to an embodiment.

[13] FIG. 2 is a schematic illustration of a simplified block diagram of a communication system according to an embodiment.

[14] FIG. 3 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment.

[15] FIG. 4 is a schematic illustration of a simplified block diagram of an encoder according to an embodiment.

[16] FIG. 5 is a schematic illustration of ARC parameter signaling embodiments in accordance with the prior art or embodiment as indicated.

[17] FIG. 6 shows an example of a syntax table according to an embodiment.

[18] FIG. 7 is a schematic illustration of a computer system according to an embodiment.

[19] FIG. 8 shows an example of a prediction structure for adaptive resolution scalability.

[20] FIG. 9 shows an example of a syntax table according to an embodiment.

[21] FIG. 10 is a schematic illustration of a simplified flowchart for parsing and decoding a poc frame for each access unit and access unit counter value.

[22] FIG. 11 is a schematic illustration of a video bitstream structure containing multi-layer sub-images.

[23] FIG. 12 is a schematic illustration of displaying a selected sub-image with improved resolution.

[24] FIG. 13 is a flowchart of a process for decoding and displaying a video bit stream containing multi-layer sub-images.

[25] FIG. 14 is a schematic illustration of a 360° video display with a sub-image enhancement layer.

[26] FIG. 15 shows an example of sub-image layout information and its corresponding layer and image prediction structure.

[27] FIG. 16 shows an example of the layout information of a sub-image and its corresponding layer and image prediction structure with a local area spatial scalability modality.

[28] FIG. 17 shows an example of a syntax table for sub-image layout information.

[29] FIG. 18 shows an example of a SEI message syntax table for sub-image layout information.

[30] FIG. 19 shows an example syntax table for specifying output layers and profile/tier/layer information for each set of output layers.

[31] FIG. 20 shows an example syntax table for specifying an output layer mode for each set of output layers.

[32] FIG. 21 shows an example syntax table for specifying the current sub-image of each layer for each set of output layers.

[33] FIG. 22 shows an example of a syntax table for specifying a sub-image identifier.

[34] FIG. 23 shows an example of an SPS syntax table for specifying the number of sub-pictures.

[35] FIG. 24 shows an example of a PPS syntax table for specifying slice division.

[36] FIG. 25 shows an example of a PPS syntax table for specifying the number of slices in an image.

[37] FIG. 26 shows another example of a PPS syntax table for indicating the number of slices in an image with a modified syntax.

IMPLEMENTATION OF THE INVENTION

[38] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it will be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which may be embodied in various forms. However, these structures and methods can be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are presented to ensure that this description is exhaustive and complete and fully conveys the scope of legal protection to those skilled in the art. In the description, details of well-known functions and methods may be omitted to avoid unnecessary difficulty in understanding the presented embodiments.

[39] As described previously, uncompressed digital video may consist of a series of images, each image having a spatial dimension, such as 1920 x 1080 luminance samples and associated chrominance samples. A series of images can have a fixed or variable image rate (informally also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 luminance samples resolution at 60 Hz frame rate) requires a bandwidth of close to 1.5 Gbps. An hour of such video requires more than 600 GB of disk space. However, an image may be divided into one or more sub-images, and each sub-image may be further divided into one or more slices. Therefore, it may be preferable to signal by dividing a slice into a sub-image or image as a more efficient signaling method.

[40] Aspects are described herein with reference to illustrations of flowcharts and/or diagrams of methods, devices (systems), and computer-readable media in accordance with various embodiments. It should be understood that each block of the flowchart and/or diagram illustrations, as well as combinations of blocks in the block diagram and/or diagram illustrations, may be implemented using machine-readable program instructions.

[41] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. The system (100) may include at least two terminals (110-120) connected to each other via a network (150). For unidirectional data transmission, the first terminal (110) may encode video data at a local location for transmission to another terminal (120) via the network (150). The second terminal (120) may receive encoded video data of another terminal from the network (150), decode the encoded data, and display the reconstructed video data. Unidirectional data transfer may be common in media serving applications and the like.

[42] FIG. 1 shows a second pair of terminals (130, 140) providing support for bi-directional transmission of encoded video, which may occur, for example, during video conferencing. For bidirectional data transmission, each terminal (130, 140) may encode video data captured at a local location for transmission to another terminal via the network (150). Each terminal (130, 140) can also receive encoded video data transmitted by another terminal, can decode the encoded data, and can display the reconstructed video data on a local display device.

[43] In FIG. 1, the terminals (110-140) may be depicted as servers, personal computers, and smartphones, but the principles of the present disclosure cannot be limited to this. Embodiments of the present invention find use in laptop computers, tablet computers, media players, and/or specialized videoconferencing equipment. Network (150) is any number of networks that transmit encoded video data between terminals (110-140), including, for example, wired and/or wireless communication networks. The communications network (150) may communicate over circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For purposes of this discussion, the architecture and topology of the network (150) may not be essential to the operation of the present disclosure unless explained below herein.

[44] FIG. 2 illustrates, as an example of an application for the disclosed subject matter, the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, compressed video storage on digital media including CD, DVD, memory card, etc., etc. .

[45] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a digital camera, producing, for example, a stream (202) of uncompressed video samples. This sample stream (202), represented by a thick line to highlight the large amount of data compared to the encoded video bitstreams, can be processed by an encoder (203) connected to the camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (204), depicted as a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server (205) for future use. One or more streaming clients (206, 208) may access the streaming server (205) to retrieve copies (207, 209) of the encoded video bitstream (204). The client (206) may include a video decoder (210) that decodes an incoming copy of the encoded video bitstream (207) and produces an outgoing stream (211) of video samples that can be displayed on a display (212) or other rendering device (not shown). In some streaming systems, the video bitstreams (204, 207, 209) may be encoded according to certain video encoding/compression standards. Examples of these standards include ITU-T Rec. H.265. A video encoding standard, informally known as Universal Video Coding or VVC, is under development. The exposed object can be used in a VVC context.

[46] FIG. 3 may be a functional block diagram of a video decoder (210) according to one or more embodiments.

[47] The receiver (310) may receive one or more encoded video sequences to be decoded by the decoder (210); in the same or another embodiment, one encoded video sequence at a time, where the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequence may be received from a channel (312), which may be a hardware/software connection to a storage device in which the encoded video data is stored. The receiver (310) may receive the encoded video data with other data, such as encoded audio data and/or ancillary data streams, which may be sent to their respective using entities (not shown). The receiver (310) can separate the encoded video sequence from other data. To combat network jitter, a buffer memory (315) can be connected between the receiver (310) and the entropy decoder/parser (320) (“parser”). When the receiver (310) receives data from a storage/forwarding device with sufficient bandwidth and controllability or from an iso-synchronous network, the buffer memory (315) may not be needed or may be small. For use in best quality packet networks such as the Internet, a buffer (315) may be required, which may be relatively large and may preferably be of an adaptive size.

[48] The video decoder (210) may include a parser (320) for recovering symbols (321) from the entropy encoded video sequence. Categories of these symbols include information used to control the operation of the decoder (210), and potentially information to control a visualization device, such as a display (212), which is not an integral part of the decoder, but may be connected to it, as has been shown in figure 2. The control information for the rendering device(s) may be in the form of additional enhancement information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser (320) may parse/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be performed in accordance with a video encoding technology or standard and may follow principles well known to those skilled in the art, including variable length encoding, Huffman encoding, arithmetic encoding with or without context sensitivity, and so on. The parser (320) may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), and so on. An entropy decoder/parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[49] The parser (320) may perform an entropy decoding/parsing operation on the video sequence received from the buffer (315) to create symbols (321).

[50] The symbol recovery 321 may include a variety of different modules depending on the type of video image or portions thereof (such as: outer and inner image, outer and inner block) and other factors. Which modules are enabled and how can be controlled by subgroup control information parsed from the encoded video sequence using the parser (320). The flow of such subgroup control information between the parser (320) and the plurality of modules is not shown below for clarity.

[51] In addition to the functional blocks already mentioned, the decoder 210 can be conceptually divided into a number of functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with each other and can be at least partially integrated into each other. However, for purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into functional modules as follows.

[52] The first module is a scaling/inverting module (351). The scaling/inverse transform module (351) receives the quantized transform factor, as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, and so on, as symbol(s) (321) from the parser ( 320). It can output blocks containing sample values that can be entered into the aggregator (355).

[53] In some cases, the output samples of the scaling/demapping module (351) may be related to an intra-encoded block; that is, to a block that does not use prediction information from previously reconstructed images, but may use prediction information from previously reconstructed portions of the current image. Such prediction information may be provided by an intra-frame prediction module (352). In some cases, the intra-frame prediction module (352) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information extracted from the current (partially reconstructed) image (356). The aggregator (355), in some cases, adds, for each sample, prediction information generated by the intra-prediction module (352) to the output sample information provided by the upscaling/inverse transform module (351).

[54] In other cases, the output samples of the scaling/demapping module (351) may be related to an intercoding and potentially motion-compensated block. In such a case, the motion compensated prediction module (353) may access the reference image memory (357) to retrieve samples used for prediction. After motion compensation of the selected samples according to the symbols (321) associated with the block, these samples can be added by the aggregator (355) to the output of the scaler/inverter (here called residual samples or residual signal) to generate output information counts. The addresses in the reference picture memory from which the motion-compensated prediction module selects prediction samples may be controlled by motion vectors available to the motion-compensated prediction module in the form of symbols (321), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values retrieved from reference image memory, where precise subsample motion vectors, motion vector prediction mechanisms, and so on are used.

[55] The output samples of the aggregator (355) can be subjected to various loop filtering techniques in the loop filter module (356). Video compression technologies may include intra-loop filtering technologies that are controlled by parameters included in the encoded video bitstream and provided to the loop filtering module (356) as symbols (321) from the parser (320), but may also respond to meta-information received during decoding previous (in decoding order) parts of the encoded image or encoded video sequence, and also respond to sample values previously restored and filtered through loop filtering.

[56] The output of the loop filter module (356) may be a stream of samples, which may be output to the renderer (212) and also stored in the reference picture memory (356) for use in future inter-frame prediction.

[57] Certain encoded images, after complete restoration, can be used as reference images for future prediction. After the encoded picture has been completely reconstructed and the encoded picture has been identified as a reference picture (eg, by the parser (320)), the current reference picture (356) may become part of the reference picture buffer (357), and fresh current picture memory may be reallocated before starting to restore the next encoded image.

[58] Video decoder 320 may perform decoding operations according to a predefined video compression technology, which may be written in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by a video compression technology or standard in use, in the sense that it matches the syntax of the video compression technology or standard as specified in the video compression technology document or standard and, in particular, in its profiles document. It may also be necessary for compliance to ensure that the complexity of the encoded video sequence is within limits defined by the level of video compression technology or standard. In some cases, levels limit the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the layers may in some cases be further limited by hypothetical reference decoder (HRD) specifications and HRD buffer management metadata signaled in the encoded video sequence.

[59] In an embodiment, the receiver (310) may receive additional (redundant) coded video data. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (320) to correctly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[60] FIG. 4 may be a functional block diagram of a video encoder (203) according to an embodiment of the present disclosure.

[61] The encoder (203) may receive video samples from a video source (201) (which is not part of the encoder), which may capture video image(s) for encoding by the encoder (203).

[62] The video source (201) may provide the original video sequence for encoding by the encoder (203) in the form of a digital video sample stream, which may be of any suitable bit depth (eg: 8 bits, 10 bits, 12 bits, ...), any color space (eg BT.601 Y CrCB, RGB, ...) and any suitable sample structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In a multimedia serving system, the video source (201) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (203) may be a camera that captures local image information as a video sequence. Video data may be provided as many individual images that, when viewed sequentially, convey motion. The images themselves may be organized as a spatial array of pixels, in which each pixel may contain one or more samples depending on the sample structure used, color space, etc. The relationship between pixels and samples can be readily understood by one skilled in the art. The description below is oriented towards readings.

[63] In an embodiment, the encoder (203) may encode and compress images of the original video sequence into an encoded video sequence (443) in real time or with any other timing constraints as required by the application. Providing an appropriate encoding rate is one of the functions of the controller (450). The controller controls other functional modules as described below and is functionally connected to these modules. The connection is not depicted for clarity. Parameters set by the controller may include parameters related to rate control (image skip, quantizer, lambda value of rate-distortion optimization methods, ...), image size, group of picture (GOP) layout, maximum motion vector search range, and so on . Other functions of the controller (450) can be readily recognized by one skilled in the art as they may relate to a video encoder (203) optimized for a particular system design.

[64] Some video encoders operate in what one skilled in the art would readily recognize as a "coding loop". As a simplified description, the encoding loop may consist of an encoding portion of the encoder 430 (hereinafter referred to as the "source encoder") (responsible for generating symbols based on the input image to be encoded and the reference image(s)), and a (local) decoder (433) built into an encoder (203) that reconstructs the symbols to create sample data that the (remote) decoder can also create (since any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies discussed in the subject matter disclosed) . This reconstructed stream of samples is entered into the reference image memory (434). Since decoding a symbol stream produces bit-accurate results regardless of the location of the decoder (locally or remotely), the contents of the reference picture buffer are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" using prediction during decoding. This fundamental principle of reference picture synchronicity (and the resulting drift if synchronicity cannot be maintained, for example due to channel errors) is well known to one skilled in the art.

[65] The operation of the "local" decoder (433) may be the same as that of the "remote" decoder (210), which has already been described in detail above with reference to FIG. 3. Briefly referring also to FIG. 3, however, since symbols are available, and encoding/decoding of symbols into a coded video sequence by the entropy encoder (445) and parser (320) can be performed losslessly, the entropy decoding portion of the decoder (210), including the channel (312) , the receiver (310), buffer (315) and parser (320), cannot be fully implemented in the local decoder (433).

[66] At this point, the observation can be made that any decoding technology other than parsing/entropy decoding that is present in the decoder must also necessarily be present in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter of the invention focuses on the operation of the decoder. The description of encoding technologies may be shortened because they are the inverse of the fully described decoding technologies. Only some areas require more detailed description, which is given below.

[67] As part of its operation, the source encoder (430) may perform motion-compensated predictive encoding that predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence that have been designated “reference frames.” Thus, the encoding engine 432 encodes the differences between blocks of pixels of the input frame and blocks of pixels of the reference frame(s) that can be selected as the prediction reference(s) for the input frame.

[68] The local video decoder (433) may decode encoded frame video data, which may be referred to as reference frames, based on the symbols generated by the source encoder (430). The operations of the encoding engine (432) may preferably be lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can usually be a copy of the original video sequence with some errors. The local video decoder (433) copies the decoding processes that may be performed by the video decoder on the reference frames and may cause the reconstructed reference frames to be stored in a reference picture cache (434). Thus, the encoder (203) may locally store copies of reconstructed reference frames that have common content as reconstructed reference frames that will be received by the far-end video decoder (assuming no transmission errors).

[69] The predictor (435) may perform predictive searches for the encoding engine (432). That is, for a new image to be encoded, the predictor (435) may search the reference image memory (434) for sample data (as reference pixel block candidates) or certain metadata such as reference image motion vectors, block shapes, etc. further, which can serve as a suitable prediction standard for new images. The predictor (435) may operate on a block-by-pixel basis to find corresponding prediction standards. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction patterns taken from a plurality of reference images stored in the reference image memory (434).

[70] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[71] The output signals of all the above functional modules may be entropy encoded in an entropy encoder (445). The entropy encoder translates the symbols generated by the various functional modules into an encoded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, and so on.

[72] The transmitter (440) may buffer the encoded video sequence(s) generated by the entropy encoder (445) to prepare it for transmission through a communication channel (460), which may be a hardware/software connection to a storage device that will store the encoded video data . The transmitter (440) may merge the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or ancillary data streams (sources not shown).

[73] The controller (450) may control the operation of the encoder (203). During encoding, the controller (450) may assign each encoded image a specific encoded image type, which may affect the encoding techniques that can be applied to the corresponding image. For example, images can often be classified into one of the following frame types.

[74] The internal picture (I-picture) may be one that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of internal images, including, for example, images with independent decoder update. One skilled in the art is aware of these variants of I-images and their respective applications and features.

[75] A prediction picture (P picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most one motion vector and a reference index to predict the sample values of each block.

[76] A bidirectional prediction picture (B picture) may be one that can be encoded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multi-prediction images can use more than two reference images and associated metadata to reconstruct a single block.

[77] Source images can typically be spatially divided into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. Blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the encoding assignment applied to the corresponding block images. For example, I-picture blocks may be encoded without prediction, or they may be predictively encoded with reference to already encoded blocks of the same image (spatial prediction or intra-prediction). P-picture pixel blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one previously encoded reference picture. B-picture blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one or two previously encoded reference pictures.

[78] The video encoder (203) may perform encoding operations in accordance with a predefined video encoding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (203) may perform various compression operations, including predictive encoding operations, that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore follow the syntax specified by the video encoding technology or standard being used.

[79] In an embodiment, the transmitter (440) may transmit additional data with encoded video. The video encoder (430) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant images and slices, additional enhancement information (SEI) messages, video suitability information (VUI) parameter set fragments, and so on.

[80] Before describing in more detail certain aspects of the disclosed subject matter, it is necessary to introduce several terms to which reference will be made throughout the remainder of this description.

[81] Sub-image further refers to, in some cases, a rectangular arrangement of samples, blocks, macroblocks, encoding units, or similar objects that are semantically grouped and that can be independently encoded at a changed resolution. One or more sub-images may be used for an image. One or more coded sub-pictures may form a coded picture. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In certain environments, one or more encoded sub-images may be collected in a compressed region without recoding to the sample level into the encoded image, and in the same or some other cases, one or more encoded sub-images may be extracted from a encoded image in the compressed region.

[82] In the following, adaptive resolution resampling (ARC) refers to mechanisms that allow the resolution of an image or sub-image in an encoded video sequence to be changed, for example, by resampling a reference image. In the following, ARC parameters refer to the control information necessary to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, output and/or reference image resolutions, various control flags, and so on.

[83] The above description focuses on the encoding and decoding of a single semantically independent encoded video image. Before describing the consequences of encoding/decoding multiple sub-pictures with independent ARC parameters and their implied additional complexity, ARC parameter signaling options must be described.

[84] FIG. 5 shows several new variants of ARC parameter signaling. As noted for each option, they have certain advantages and certain disadvantages in terms of coding efficiency, complexity, and architecture. A video coding standard or technology may select one or more of these options or options known in the prior art for signaling ARC parameters. The options may not be mutually exclusive and may possibly be interchangeable depending on the needs of the application, the standards used, or the choice of encoder.

[85] ARC parameter classes may include the following.

[86] - Upsampling/downsampling factors, individual or combined in X and Y dimensions.

[87] - Upsampling/downsampling factors with added time dimension, indicating upsampling/downsampling at a constant rate for a given number of images.

[88] - Either of the above two options may involve encoding one or more presumably short syntax elements that may indicate a table containing the coefficient(s).

[89] - Resolution in the X or Y dimension in units of samples, blocks, macroblocks, CUs, or any other suitable granularity of the input image, output image, reference image, coded image, in combination or individually. If there is more than one resolution (for example, one for the input image, one for the reference image), then in some cases one set of values can be inferred from another set of values. This can be communicated, for example, by using flags. See below for a more detailed example.

[90] - "Warp" coordinates similar to those used in Annex P of H.263, again with a suitable level of detail as described above. Annex P H.263 specifies one efficient way to encode such deformation coordinates, but other, potentially more efficient ways may also be being developed. For example, reverse Huffman coding with variable length warp coordinates according to Appendix P can be replaced by binary coding of suitable length, where the length of the binary codeword can be, for example, derived from the maximum image size, perhaps multiplied by a certain factor and offset by a certain value, so that take into account “deformation” beyond the limits of the maximum image size.

[91] - Upsampling and/or downsampling filter options. In the simplest case, there may be only one filter for upsampling and/or downsampling. However, in some cases it may be preferable to provide more flexibility in the filter design, and this may require signaling the filter parameters. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g. by a list of filter coefficients using suitable entropy coding techniques), the filter may be implicitly selected by appropriate upsampling/downsampling ratios that , in turn, are signaled according to any of the mechanisms mentioned above, and so on.

[92] The following description assumes encoding a finite set of upsampling/downsampling factors (the same factor that should be used in the X and Y dimensions) specified by a codeword. This codeword may preferably be encoded with a variable length, for example using exponential Golomb code, common to certain syntax elements in video encoding specifications such as H.264 and H.265. One suitable mapping of values to upsampling/downsampling factors could, for example, correspond to the following table.

A codeword Exponential Golomb code Source/Target Resolution 0 1 1/1 1 010 1/1.5 (50% increase) 2 011 1.5/1 (50% reduction) 3 00100 1/2 (100% increase) 4 00101 2/1 (100% reduction)

[93] Many similar mappings can be developed according to the needs of the application and the capabilities of the upsampling and downsampling mechanisms available in the video compression technology or standard. The table can be expanded to include more values. Values can also be represented by entropy encoding mechanisms other than Exponential Golomb codes, such as using binary encoding. This may have certain advantages where resampling factors were of interest outside of the video processing engines themselves (primarily the encoder and decoder), for example through MANE. It should be noted that for the (presumably) most common case where no resolution change is required, the Exponential Golomb code, which is short, can be chosen; there is only one bit in the table above. This may have an advantage in coding efficiency over using binary codes for the most common case.

[94] The number of records in the table, as well as their semantics, can be fully or partially customized. For example, the underlying table structure may be conveyed in a "high" parameter set, such as a sequence or decoder parameter set. Alternatively or in addition, one or more such tables may be defined in a video encoding technology or standard and may be selected, for example, by a decoder or sequence parameter set.

[95] The following describes how the upsampling/downsampling factor (ARC information) encoded as described above can be included in a video encoding technology or standard syntax. Similar considerations may apply to one or more codewords driving the upsampling/downsampling filters. See the discussion below when a filter or other data structures require relatively large amounts of data.

[96] H.263 Annex P includes ARC information 502 in the form of four warp coordinates in the image header 501, specifically in the H.263 PLUSPTYPE (503) header extension. This may be a reasonable design choice when a) the image header is available and b) frequent changes to the ARC information are expected. However, the overhead when using H.263 style signaling can be quite high, and the scaling factors may not apply to image boundaries since the image header may be temporary.

[97] JVCET-M135-v1, cited above, includes ARC reference information (505) (index) located in the image parameter set (504), index table (506), including target resolutions, which in turn is located within set (507) of sequence parameters. Placing a possible resolution in a table (506) in a sequence parameter set (507) may, in accordance with verbal statements made by the authors, be justified by using the SPS as a point of interoperability negotiation during capability exchange. The resolution can be varied within the limits set by the values in the table (506) from image to image by accessing the appropriate set (504) of image parameters.

[98] Still referring to FIG. 5, the following additional options may exist for transmitting ARC information in the video bit stream. Each of these options has certain advantages over the current state of the art, as described above. Options can be present simultaneously in the same video encoding technology or standard.

[99] In an embodiment, ARC information (509), such as a resampling (scaling) factor, may be present in a slice header, GOB header, tile header, or tile group header (hereinafter, tile group header) (508). This may be sufficient for small size ARC information such as a single variable length codeword ue(v) or a fixed length codeword of several bits such as shown above. Having ARC information directly in the tile group header has the added benefit that the ARC information can be applied to a sub-image represented by, for example, that tile group rather than the entire image. See also below. Additionally, even if a video compression technology or standard only provides for changing the adaptive resolution of the entire image (as opposed to, for example, adaptive resolution changes on a per-tile group basis), placing the ARC information in the tile group header instead of placing it in the H. 263 has certain advantages in terms of error tolerance.

[100] In the same or another embodiment, the ARC information (512) itself may be present in a corresponding parameter set (511), such as, for example, an image parameter set, a title parameter set, a tile parameter set, an adaptation parameter set, and so on ( shows a set of adaptation parameters). The scope of this parameter set may preferably be no larger than an image, such as a group of tiles. The use of ARC information is implemented implicitly by activating the appropriate set of parameters. For example, when a video encoding technology or standard considers only image-based ARC, then an image parameter set or equivalent may be an appropriate option.

[101] In the same or another embodiment, ARC reference information (513) may be present in a tile group header (514) or similar data structure. This reference information (513) may refer to a subset of ARC information (515) available in a parameter set (516) with a scope beyond a single image, such as a sequence parameter set or a decoder parameter set.

[102] The additional level of indirection involving activation of PPS from the tile group header, PPS, SPS, which is used in JVET-M0135-v1, seems unnecessary since image parameter sets, like sequence parameter sets, can be used (and are used in some standards such as RFC3984) to negotiate capabilities or declarations. However, if the ARC information is to be applied to a sub-image represented, for example, by groups of tiles as well, a better choice may be a parameter set with an activation region limited to the group of tiles, such as an adaptation parameter set or a header parameter set. Also, if the ARC information is more than negligible in size - for example, containing filter control information such as multiple filter coefficients - then the parameter may be a better choice than using the header (508) directly in terms of encoding efficiency, since these settings may be reused in future images or subimages, referencing the same set of parameters.

[103] When using a sequence parameter set or other higher level parameter set with a scope spanning multiple images, the following specific considerations may apply.

[104] 1. The set of parameters for storing the ARC information table (516) may in some cases be a set of sequence parameters, but in other cases preferably a set of decoder parameters. A set of decoder parameters may have an activation region of several CVS, namely the encoded video stream, i.e. all encoded bits of the video from the beginning to the end of the session. This scope may be more appropriate because the possible ARC coefficients may be a function of the decoder, perhaps implemented in hardware, and hardware functions generally do not change with any CVS (which, at least in some entertainment systems, is a group images one second or less in length). However, placing a table in the sequence options set is explicitly included in the placement options described here, particularly in relation to point 2 below.

[105] 2. The ARC reference information (513) may preferably be placed directly in the image/tile slice/GOB/tile group header (hereinafter referred to as the tile group header) (514), rather than in the image parameter set as in JVCET-M0135- v1. The reason is this: when the encoder wants to change one value in the picture parameter set, such as the ARC reference information, it must create a new PPS and reference that new PPS. Suppose that only the ARC reference information changes, but other information, such as, for example, the quantization matrix information in the PPS, remains. Such information may be of considerable size and will need to be retransmitted in order to complete the new PPS. Since the ARC reference information may be a single codeword, such as an index in the table (513), and it will be the only value that changes, it would be cumbersome and wasteful to retransmit, for example, all the quantization matrix information. In this regard, it may be significantly better in terms of coding efficiency to avoid indirection via PPS, as proposed in JVET-M0135-v1. Similarly, placing the ARC reference information in the PPS has the additional disadvantage that the ARC information referenced by the ARC reference information (513) must necessarily apply to the entire image and not to a sub-image, since the activation volume of the image parameter set represents is an image.

[106] In the same or another embodiment, ARC parameter signaling may follow a detailed example as shown in FIG. 6. 6 shows syntax diagrams in the representation used in video coding standards since at least 1993. The notation of such syntax diagrams roughly corresponds to C-style programming. Lines in bold indicate syntax elements present in the bitstream; lines without bold often indicate control flow or setting variables.

[107] The tile group header (601), as an exemplary syntactic structure of the header applied to the (possibly rectangular) portion of the image, may conventionally contain an exponential Golomb encoded variable length syntax element dec_pic_size_idx (602) (in bold). The presence of this syntax element in the tile group header can be limited by the use of adaptive resolution (603) - here the flag value is not in bold, meaning that the flag is present in the bitstream at the point where it occurs in the syntax diagram. The fact whether adaptive resolution is used for this image or parts thereof can be signaled in any high-level syntactic structure inside or outside the bitstream. In the example shown, this is signaled in the sequence parameter set as shown below.

[108] Still referring to FIG. 6, a sample of a set (610) of sequence parameters is also shown. The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, this flag may indicate the use of adaptive resolution, which in turn may require certain control information. In the example, such control information is present conditionally based on the value of a flag based on the if() statement in the parameter set (612) and the tile group header (601).

[109] When adaptive resolution is used, this example encodes the output resolution in sample units (613). Item number 613 refers to both output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can determine the resolution of the output image. Elsewhere in the video encoding technology or standard, certain limits may be specified for any value. For example, the level definition may limit the number of total output samples that can be the product of the values of these two syntax elements. Also, certain video encoding technologies or standards, or external technologies or standards, such as system standards, may limit the numbering range (for example, one or both dimensions must be divisible by a power of 2) or the width-to-height ratio (for example, width and height should be in a ratio such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementations or for other reasons, and are well known in the art.

[110] In some applications, it may be desirable for the encoder to instruct the decoder to use a certain reference image size, rather than implicitly assuming that this size is the output image size. In this example, the syntax element reference_pic_size_present_flag (614) conveys the conditional presence of the dimensions (615) of the reference picture (again, the position number refers to both width and height).

[111] Finally, a table of possible decoding image widths and heights is shown. Such a table can be expressed, for example, by specifying the table (num_dec_pic_size_in_luma_samples_minus1) (616). "Minus1" may refer to the interpretation of the meaning of this syntax element. For example, if the encoded value is zero, there is one entry in the table. If the value is five, there are six table entries. Then, for each "row" in the table, the width and height of the decoded image are included in the syntax (617).

[112] Presented table entries (617) may be indexed using the dec_pic_size_idx syntax element (602) in the tile group header, thereby allowing different decoded sizes—essentially, scaling factors—for each tile group.

[113] Some video encoding technologies or standards, such as VP9, support spatial scalability by implementing certain forms of reference image resampling (signaled entirely differently than in the disclosed subject matter) in combination with temporal scalability to provide spatial scalability. In particular, some reference images may be upsampled using ARC-style techniques to a higher resolution to form the base of the spatial enhancement layer. These upsampled images can be enhanced by using normal high-resolution prediction engines to add detail.

[114] The disclosed subject matter may be used in such an environment. In some cases, in the same or another embodiment, a value in the NAL unit header, such as a temporal identifier field, may be used to indicate not only a temporal layer, but also a spatial layer. This provides certain benefits for certain system configurations; for example, the existing Selected Forwarding Units (SFU) architecture, created and optimized for selected temporal layers forwarded based on the value of the NAL unit's temporary header identifier, can be used without modification for scale-out environments. To make this possible, there may be a requirement for a mapping between the encoded image size and the temporal layer indicated by the temporary identifier field in the NAL unit header.

[115] In some video encoding technologies, an access unit (AU) may refer to the encoded image(s), slice(s), tile(s), NAL block(s), and so on, that have been captured and arranged into a corresponding image/ slice/tile/bitstream of a NAL unit at a given time. Such a point in time could be the time of composition.

[116] In HEVC and some other video encoding technologies, a picture order counter (POC) value can be used to indicate a selected reference picture among a plurality of reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) contains one or more images, slices or tiles, each image, slice or tile belonging to the same AU may carry the same POC value, from which it can be inferred that it was created from content at the same layout time. In other words, in a scenario where two images/slices/tiles carry the same set POC value, it may indicate that the two images/slices/tiles belong to the same AU and have the same layout time. Conversely, two images/slices/tiles having different POC values may indicate those images/slices/tiles that belong to different AUs and have different composition times.

[117] In an embodiment of the disclosed subject matter, the above-mentioned tight relationship may be relaxed since the access unit may contain images, slices, or tiles with different POC values. By allowing different POC values in the AU, it becomes possible to use the POC value to identify potentially independently decodable pictures/slices/tiles with identical presentation times. This, in turn, can support multiple scalable layers without changing reference picture selection signaling (eg, reference picture set signaling or reference picture list signaling), as described in more detail below.

[118] However, it is still desirable to be able to identify the AU that owns an image/slice/tile with respect to another image/slice/tile having different POC values based only on the POC value. This can be achieved as described below.

[119] In the same or other embodiments, an access unit counter (AUC) may be signaled in a high-level syntactic structure such as a NAL unit header, a slice header, a tile group header, an SEI message, a parameter set, or an AU delimiter. The AUC value can be used to identify which NAL units, images, slices or tiles belong to a given AU. The AUC value may correspond to a single point in time of the composition. The AUC value can be a multiple of the POC value. By dividing the POC value by an integer value, the AUC value can be calculated. In some cases, division operations may impose some overhead on decoder implementations. In such cases, slight restrictions in the numbering space of AUC values may allow the division operation to be replaced by shift operations. For example, the AUC value may be equal to the value of the most significant bit (MSB) of the POC value range.

[120] In the same embodiment, the POC cycle value on the AU (poc_cycle_au) may be signaled in a high-level syntactic structure such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or AU delimiter. The poc_cycle_au value can indicate how many different and consecutive POC values can be associated with the same AU. For example, if poc_cycle_au is 4, images, slices, or tiles with a POC value of 0-3 inclusive are associated with an AU with an AUC value of 0, and images, slices, or tiles with a POC value of 4-7, inclusive, are associated with an AU with an AUC value of 1. Therefore, the AUC value can be derived by dividing the POC value by the poc_cycle_au value.

[121] In the same or another embodiment, the poc_cyle_au value may be derived from information located, for example, in a video parameter set (VPS) that identifies the number of spatial layers or SNR layers in the encoded video sequence. This possible relationship is briefly described below. Although obtaining as described above may save a few bits in the VPS and therefore may improve encoding efficiency, it may be preferable to explicitly encode poc_cycle_au in an appropriate high-level syntactic structure hierarchically below the video parameter set to be able to minimize poc_cycle_au for a given small portion of the bit stream such as an image. This optimization can save more bits than can be stored by the acquisition process described above because POC values (and/or syntax element values indirectly related to POC) can be encoded in low-level syntactic structures.

[122] The methods for signaling adaptive resolution parameters described above may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 7 illustrates a computer system 700 suitable for implementing certain embodiments of the disclosed subject matter.

[123] Computer software may be encoded using any suitable machine code or computer language that may be subject to assembly, compilation, linking or similar mechanisms to create code containing instructions that can be executed directly or through interpretation, execution of microcode, etc. etc., computer central processing units (CPU), graphic processing units (GPU), etc.

[124] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[125] The components shown in FIG. 7 for computer system 700 are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of components should also not be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the exemplary embodiment of computer system 700.

[126] The computer system 700 may include certain human-machine interface input devices. Such a human-machine interface input device may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, control glove movement), audio input (e.g., voice, hand clapping), visual input (for example: gestures), olfactory input (not shown). Human-machine interface devices can also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, ambient sound), images (e.g. scanned images, photographic images captured from a camera for still images), video (e.g. 2D video, 3D video, including stereoscopic video).

[127] Human-machine interface input devices may include one or more of (only one of each illustrated): keyboard 701, mouse 702, trackpad 703, touch screen 710, control glove 704, joystick 705, microphone 706, scanner 707, camera 708.

[128] The computer system 700 may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human-machine interface output devices may include haptic output devices (e.g., haptic feedback from touch screen 710, control glove 704, or joystick 705, but there may also be haptic feedback devices that do not serve as input devices) , audio output devices (eg, speakers 709, headphones (not shown)), image output devices (eg, screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have touch screen input, each with or without haptic feedback capability - some of which may be capable of providing two-dimensional visual output or more than three-dimensional output through means such as stereographic output; virtual reality glasses (not shown), holographic displays and smoke tanks (not pictured)) and printers (not pictured).

[129] Computer system 700 may also include human accessible storage devices and associated media, such as optical media, including CD/DVD ROM/RW 720 with CD/DVD or similar media 721, flash drive 722, removable hard disk drive or solid state drive 723, previously developed magnetic media such as tape and floppy disk (not shown), specialized ROM/ASIC/PLD based devices such as dongles (not shown), etc.

[130] Those skilled in the art will also understand that the term "machine readable medium" as used in connection with the subject matter disclosed herein does not cover transmission media, carrier waves, or other time signals.

[131] Computer system 700 may also include an interface to one or more communications networks. Networks can be, for example, wireless, wired, optical. Additionally, networks can be LAN, WAN, metropolitan, automotive and industrial, real-time, latency tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired TV or wireless wide area digital networks, including cable TV, satellite TV and terrestrial TV, automotive and industrial equipment, including CAN bus, etc. Some networks usually require external network interface adapters that are connected to certain general purpose data ports or peripheral buses (749) (for example, to the USB ports of the computer system 700); others are typically integrated into the core of the computer system 700 by connecting to a system bus, as described below (for example, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 700 can communicate with other entities. Such communication may be unidirectional, receive only (for example, broadcast television), unidirectional, send only (for example, CAN bus to certain CAN bus devices), or bidirectional, such as for other computer systems using a local or wide area digital network . Each of these networks and network interfaces may use specific protocols and protocol stacks, as described above.

[132] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core 740 of the computer system 700.

[133] Core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, specialized programmable processors in the form of field programmable gate arrays (FPGAs) 743, task-specific hardware accelerators 744, etc. These devices, along with read-only memory (ROM) 745, random access memory 746, internal mass storage such as internal hard drives not accessible to the user, solid-state drives (SSD), etc. 747, can be connected through the system bus 748. On some computer systems, the system bus 748 may be available as one or more physical connectors to allow expansion with additional CPUs, GPUs, etc. Peripherals may be connected either directly to the kernel system bus 748 or through peripheral bus 749. Peripheral bus architecture includes PCI, USB, etc.

[134] The CPU 741, GPU 742, FPGA 743, and accelerators 744 may execute certain instructions that, in combination, may constitute the aforementioned computer code. This computer code may be stored in ROM 745 or RAM 746. Transient data may also be stored in RAM 746, while persistent data may be stored, for example, in internal mass storage 747. Rapid storage and retrieval to either storage device may be accessible by using a cache memory that may be closely coupled to one or more of a CPU 741, a GPU 742, a mass storage device 747, a ROM 745, a RAM 746, or the like.

[135] A computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specifically designed and constructed for the purposes of the present disclosure, or they may be well known and available to those skilled in the art of computer software.

[136] By way of example, and not limitation, a computer system having architecture 700, and in particular core 740, may provide functionality as a result of having processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied on one or more tangible, machine-readable media. Such computer-readable media may be media associated with user-accessible mass storage, as described above, as well as certain core 740 storage devices that are non-transitory in nature, such as internal mass storage 747 or ROM 745. Software implementing various embodiments implementation of the present disclosure, may be stored in such devices and executed by core 740. A computer-readable medium may include one or more storage devices or chips according to specific needs. Software may cause the core 740 and, in particular, processors therein (including CPUs, GPUs, FPGAs, and the like) to execute certain processes or certain portions of specific processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to processes defined by the software. In addition or alternatively, a computer system may provide functionality as a result of logic built into hardware circuitry or otherwise embodied in circuitry (e.g., accelerator 744), which may operate instead of or in conjunction with software to perform certain processes or portions of specific processes described in this document. A reference to software may include logic and vice versa where applicable. Reference to a computer-readable medium may include circuitry (eg, an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both, where applicable. The present disclosure includes any suitable combination of hardware and software.

[137] FIG. 8 shows an example of a video sequence structure with a combination of temporal_id, layer_id, POC and AUC values with adaptive resolution changing. In this example, an image, slice, or tile in the first AU with AUC = 0 may have temporal_id = 0 and layer_id = 0 or 1, while an image, slice, or tile in the second AU with AUC = 1 may have temporal_id = 1 and layer_id = 0 or 1 respectively. The POC value is incremented by 1 for each image, regardless of the temporal_id and layer_id values. In this example, the value of poc_cycle_au could be equal to 2. Preferably, the value of poc_cycle_au could be set equal to the number of layers (spatial scalability). Therefore, in this example, the POC value is increased by 2 and the AUC value is increased by 1.

[138] In the above embodiments, all or a subset of the inter-frame or inter-layer prediction structure and reference picture indication may be supported using existing HEVC Reference Picture Set (RPS) signaling or Reference Picture List (RPL) signaling. In RPS or RPL, the selected reference picture is indicated by signaling a POC value or a POC delta value between the current picture and the selected reference picture. For the disclosed subject matter, RPS and RPL can be used to indicate an inter-frame or inter-layer prediction structure without changing the signaling, but with the following limitations. If the temporal_id value of the reference picture is greater than the temporal_id value of the current picture, the current picture may not use the reference picture for motion compensation or other predictions. If the layer_id value of the reference image is greater than the layer_id value of the current image, the current image may not use the reference image for motion compensation or other predictions.

[139] In the same and other embodiments, POC difference-based motion vector scaling for temporal motion vector prediction may be disabled for multiple images in an access unit. Therefore, although each image may have a different POC value in an access unit, the motion vector is not scaled and is used to temporally predict the motion vector in an access unit. This is because a reference picture with a different POC in the same AU is considered a reference picture having the same point in time. Therefore, in an embodiment, the motion vector scaling function may return 1 when the reference image belongs to an AU associated with the current image.

[140] In the same and other embodiments, motion vector scaling based on the POC difference for temporal motion vector prediction can be optionally disabled for multiple images when the spatial resolution of the reference image differs from the spatial resolution of the current image. When motion vector scaling is enabled, the motion vector is scaled based on both the POC difference and the spatial resolution ratio between the current image and the reference image.

[141] In the same or another embodiment, the motion vector may be scaled based on the AUC difference instead of the POC difference to temporally predict the motion vector, especially when poc_cycle_au has a non-uniform value (when vps_contant_poc_cycle_per_au == 0). Otherwise (when vps_contant_poc_cycle_per_au == 1), scaling the motion vector based on the AUC difference may be identical to scaling the motion vector based on the POC difference.

[142] In the same or another embodiment, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU (with the same AUC value) with the current image is not scaled based on the AUC difference and is used to predict the motion vector without scaling or scaling based on the spatial resolution ratio between the current image and the reference image.

[143] In the same and other embodiments, the AUC value is used to identify the boundary of the AU and is used for a hypothetical reference decoder (HRD) operation that requires input and output synchronization to the AU granularity. In most cases, the decoded image with the highest level in the AU can be output for display. The AUC value and layer_id value can be used to identify the output image.

[144] In an embodiment, an image may consist of one or more sub-images. Each sub-image can cover a local area or the entire image area. The area supported by a sub-image may or may not overlap with the area supported by another sub-image. The area composed of one or more sub-images may or may not cover the entire image area. If an image consists of a sub-image, the area supported by the sub-image is identical to the area supported by the image.

[145] In the same embodiment, the sub-image may be encoded with an encoding method similar to the encoding method used for the encoded image. The sub-image may be independently encoded or may be encoded depending on another sub-image or encoded image. A sub-image may or may not have any parsing dependency on another sub-image or encoded image.

[146] In the same embodiment, the encoded sub-image may be contained in one or more layers. The encoded sub-image in a layer may have a different spatial resolution. The original sub-image may be spatially resampled (upsampled or downsampled), encoded at different spatial resolution parameters, and contained in a bitstream corresponding to the layer.

[147] In the same or another embodiment, a sub-image with ( W , H ), where W denotes the width of the sub-image and H denotes the height of the sub-image, respectively, may be encoded and contained in the encoded bitstream corresponding to layer 0, while the sub-image upsampled (or downsampled) from a subimage with the original spatial resolution, c ( W*S _w,k , H*S _h,k ), may be encoded and contained in the encoded bitstream corresponding to layer k, where S _w,k , S _h,k indicate the horizontal and vertical oversampling coefficients. If the values of S _w,k , S _h,k are greater than 1, oversampling is equal to upsampling. Whereas, if the values of S _w,k, S _h,k are less than 1, oversampling is equal to downsampling.

[148] In the same or another embodiment, an encoded sub-image in a layer may have a different visual quality than an encoded sub-image in another layer in the same sub-image or another sub-image. For example, sub-image i in layer n is encoded by the quantization parameter Q _i,n, while sub-image j in layer m is encoded by the quantization parameter Q _j,m .

[149] In the same or another embodiment, an encoded sub-image in a layer can be decoded independently, without any dependence on parsing or decoding, from an encoded sub-image in another layer of the same local area. A sub-image layer that can be independently decoded without reference to another sub-image layer of the same local region is an independent sub-image layer. An encoded subimage in an independent subimage layer may or may not have a decoding or parsing dependency on a previously encoded subimage in the same subimage layer, but an encoded subimage may not have any dependency on an encoded image in another subimage layer.

[150] In the same or another embodiment, an encoded sub-image in a layer may be decoded dependently, with any dependence on parsing or decoding, from an encoded sub-image in another layer of the same local area. A sub-image layer that can be dependently decoded with reference to another sub-image layer of the same local region is a dependent sub-image layer. An encoded subimage in a dependent subimage may reference an encoded subimage belonging to the same subimage, a previously encoded subimage in the same subimage layer, or both reference subimages.

[151] In the same or another embodiment, the encoded sub-image consists of one or more independent sub-image layers and one or more dependent sub-image layers. However, at least one independent sub-picture may be present for the encoded sub-picture. An independent sub-image layer may have a layer identifier (layer_id) value, which may be present in the NAL unit header or other high-level syntax structure, equal to 0. A sub-image layer with a layer_id equal to 0 is the base sub-image layer.

[152] In the same or another embodiment, the image may consist of one or more foreground sub-images and one background sub-image. The area supported by the background subimage may be equal to the image area. The region supported by the foreground sub-image may overlap with the region supported by the background sub-image. The background subimage may be a base layer of the subimage, while the foreground subimage may be a non-baseline (enhancement) layer of the subimage. One or more non-base layers of a sub-image may reference the same base layer for decoding. Each non-base subimage layer with layer_id equal to a can reference a non-base subimage layer with layer_id equal to b , where a is greater than b .

[153] In the same or another embodiment, the image may consist of one or more foreground sub-images with or without a background sub-image. Each sub-image may have its own base sub-image layer and one or more non-base (enhancement) layers. Each sub-image base layer may be referenced by one or more non-base sub-image layers. Each non-base subimage layer with layer_id equal to a can reference a non-base subimage layer with layer_id equal to b , where a is greater than b .

[154] In the same or another embodiment, the image may consist of one or more foreground sub-images with or without a background sub-image. Each encoded sub-image in a (base or non-base) sub-image layer may be referenced by one or more non-base layer sub-images that belong to the same sub-image, and one or more non-base layer sub-images that do not belong to the same sub-image. .

[155] In the same or another embodiment, the image may consist of one or more foreground sub-images with or without a background sub-image. The sub-image in layer a can be further divided into multiple sub-images in the same layer. One or more encoded sub-pictures in layer b may reference divided sub-pictures in layer a .

[156] In the same or another embodiment, a coded video sequence (CVS) may be a group of coded pictures. A CVS may consist of one or more coded sub-picture sequences (CSPS), where a CSPS may be a group of coded sub-pictures covering the same local image region. CSPS may have the same or a different temporal resolution as the encoded video sequence.

[157] In the same or another embodiment, the CSPS may be encoded and contained in one or more layers. The CSPS may consist of one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS can reconstruct a sequence of sub-images corresponding to the same local region.

[158] In the same or another embodiment, the number of CSPS layers corresponding to a CSPS may be identical to or different from the number of CSPS layers corresponding to another CSPS.

[159] In the same or another embodiment, a CSPS layer may have a different temporal resolution (eg, frame rate) than another CSPS layer. The original (uncompressed) subimage sequence may be temporally resampled (upsampled or downsampled), encoded at different temporal resolution parameters, and contained in a bitstream corresponding to the layer.

[160] In the same or another embodiment, the sub-image sequence at frame rate F may be encoded and contained in the encoded bitstream corresponding to layer 0, while the sub-image sequence is temporally upsampled (or down-sampled) from the original sub-image sequence at F *S _t,k may be encoded and contained in a coded bitstream corresponding to layer k _, where S _t,k indicates the time sampling rate for layer k. If the value of S _t,k is greater than 1, the temporal resampling process is equal to frame rate upconversion. Whereas, if the value of S _t,k is less than 1, the temporal resampling process is equal to frame rate down-conversion.

[161] In the same or another embodiment, when the sub-image with CSPS layer a is a reference of the sub-image with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b , the decoded pixels in the layer a CSPS are resampled and used for reference. The resampling process may require upsampling or downsampling filtering.

[162] FIG. 9 shows example syntax tables for signaling the vps_poc_cycle_au syntax element in a VPS (or SPS), which specifies the poc_cycle_au used for all pictures/slices in the encoded video sequence, and the slice_poc_cycle_au syntax element, which specifies the poc_cycle_au of the current slice in the slice header. . If the POC value increases uniformly for each AU, vps_contant_poc_cycle_per_au in the VPS is set to 1 and vps_poc_cycle_au is signaled in the VPS. In this case, slice_poc_cycle_au is not explicitly signaled and the AUC value for each AU is calculated by dividing the POC value by vps_poc_cycle_au. If the POC value does not increase uniformly per AU, vps_contant_poc_cycle_per_au in the VPS is set to 0. In this case, vps _access_unit_cnt is not signaled, while slice_access_unit_cnt is signaled in the slice header for each slice or image. Each slice or image can have a different slice_access_unit_cnt value. The AUC value for each AU is calculated by dividing the POC value by slice_poc_cycle_au. FIG. 10 is a flowchart illustrating a corresponding workflow.

[163] In the same or other embodiments, even though the POC value of an image, slice, or tile may be different, an image, slice, or tile corresponding to an AU with the same AUC value may be associated with the same decoding or output timing . Therefore, without any dependency on interparsing and/or decoding between images, slices or tiles in the same AU, all or a subset of images, slices or tiles associated with the same AU can be decoded in parallel and can be released at the same point in time.

[164] In the same or other embodiments, even though the POC value of an image, slice, or tile may be different, an image, slice, or tile corresponding to an AU with the same AUC value may be associated with the same layout/display time . When the layout time is contained in the container format, even if the images correspond to different AUs, as long as the images have the same layout time, the images can be displayed at the same point in time.

[165] In the same or other embodiments, each image, slice, or tile may have the same temporal identifier (temporal_id) in the same AU. All or a subset of images, slices or tiles corresponding to a point in time can be associated with the same time sublayer. In the same or other embodiments, each image, slice, or tile may have the same or a different spatial layer identifier (layer_id) in the same AU. All or a subset of images, slices, or tiles corresponding to a point in time can be associated with the same or a different spatial layer.

[166] FIG. 11 shows an example video stream including a background CSPS video with a layer_id of 0 and a plurality of foreground CSPS layers. Although an encoded sub-image may be composed of one or more CSPS layers, a background region that does not belong to any foreground CSPS layer may be composed of a base layer. The base layer may contain a background region and a foreground region, while the CSPS enhancement layer contains a foreground region. The CSPS enhancement layer may have better visual quality than the base layer in the same area. The CSPS enhancement layer may reference the reconstructed pixels and motion vectors of the base layer corresponding to the same region.

[167] In the same or another embodiment, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers corresponding to each sub-image are contained in a separate track in the video file.

[168] In the same or another embodiment, the video bitstream corresponding to the base layer is contained in a track, while CSPS layers with the same layer_id are contained in a separate track. In this example, the track corresponding to layer k includes only the CSPS layers corresponding to layer k .

[169] In the same or another embodiment, each CSPS layer of each sub-image is stored in a separate track. Each track may or may not have a parsing or decoding dependency on one or more other tracks.

[170] In the same or another embodiment, each track may contain bitstreams corresponding to layer i to layer j of the CSPS layers of all or a subset of sub-images, where 0 <i=j=k, k is the layer of the largest CSPS layer.

[171] In the same or another embodiment, the image consists of one or more associated media data, including a depth map, alpha map, 3D geometry data, occupancy map, etc. Such associated synchronized media data may be divided into one or multiple sub-data streams, each corresponding to one sub-image.

[172] In the same or another embodiment, FIG. 12 shows an example of a video conference based on a multi-layer sub-image method. The video stream contains one base layer video bitstream corresponding to a background image and one or more enhancement layer video bitstreams corresponding to foreground sub-images. Each video bitstream of the enhancement layer corresponds to a CSPS layer. The default display shows the image corresponding to the base layer. It contains one or more users' picture in picture (PIP). When a particular user is selected by the client control, the CSPS enhancement layer corresponding to the selected user is decoded and displayed with improved quality or spatial resolution. Fig. 13 shows a flowchart of operation.

[173] In the same or another embodiment, a network intermediate unit (such as a router) may select a subset of layers to send to the user depending on its bandwidth. The image/sub-image organization can be used to adapt the bandwidth. For example, if the user does not have the bandwidth, the router splits the layers or selects some sub-images based on their importance or based on the settings used, and this can be done dynamically to adapt to the bandwidth.

[174] FIG. 14 shows an example of using 360-degree video. When a 360 degree spherical image is projected onto a flat image, the 360 projection image can be divided into many sub-images as a base layer. The enhancement layer of a particular sub-image may be encoded and transmitted to the client. The decoder can decode both a base layer including all sub-images and an enhancement layer of a selected sub-image. When the current viewport is identical to the selected sub-image, the displayed image may have a higher quality with the decoded sub-image with an enhancement layer. Otherwise, the decoded image with the base layer may be displayed in poor quality.

[175] In the same or another embodiment, any layout information for display may be present in the file as additional information (eg, SEI messages or metadata). One or more decoded sub-pictures may be moved and displayed depending on the signaled layout information. The layout information may be signaled by the streaming server or broadcaster, or may be reconstructed by the network entity or cloud server, or may be determined by the user's individual settings.

[176] In an embodiment, when the input image is divided into one or more (rectangular) sub-regions, each sub-region may be encoded as an independent layer. Each independent layer corresponding to a local area can have a unique layer_id value. For each independent layer, information about the size and location of the sub-image can be signaled. For example, image size (width, height), information about the offset of the upper left corner (x_offset, y_offset). FIG. 15 shows an example of a split image layout, size and position information of its sub-images, and its corresponding image prediction structure. Layout information, including sub-image size(s) and sub-image position(s), may be signaled in a high-level syntactic structure such as parameter set(s), a slice or tile group header, or an SEI message.

[177] In the same embodiment, each sub-image corresponding to an independent layer may have its own unique POC value within the AU. When a reference image among the images stored in the DPB is specified using syntax element(s) in an RPS or RPL structure, the POC value(s) of each sub-image corresponding to the layer may be used.

[178] In the same or another embodiment, to indicate the (inter-layer) prediction structure, the layer_id may not be used and the POC (delta) value may be used.

[179] In the same embodiment, a sub-image with a POC value equal to N corresponding to a layer (or local region) may or may not be used as a reference image of a sub-image with a POC value equal to N+K corresponding to the same layer (or that same local area) for motion compensated prediction. In most cases, the value of the number K can be equal to the maximum number of (independent) layers, which can be identical to the number of subregions.

[180] In the same or another embodiment, FIG. 16 shows an expanded example of FIG. 15. When the input image is divided into multiple (eg, four) sub-regions, each local region may be encoded with one or more layers. In this case, the number of independent layers may be equal to the number of sub-regions, and one or more layers may correspond to a sub-region. Thus, each sub-region may be encoded by one or more independent layers and zero or more dependent layers.

[181] In the same embodiment shown in FIG. 16, the input image can be divided into four sub-regions. The top right sub-region can be encoded as two layers, which are layer 1 and layer 4, while the bottom right sub-region can be encoded as two layers, which are layer 3 and layer 5. In this case, layer 4 can refer to layer 1 for motion compensated prediction, while layer 5 can reference layer 3 for motion compensation.

[182] In the same or another embodiment, intra-loop filtering (eg, deblocking filtering, adaptive intra-loop filtering, shapeshifting, two-way filtering, or any deep learning-based filtering) across a layer boundary can be (optionally) disabled.

[183] In the same or another embodiment, motion compensated prediction or intra-block copying across a layer boundary may be (optionally) disabled.

[184] In the same or another embodiment, boundary padding for motion-compensated prediction or intra-loop filtering at the sub-image boundary may be processed optionally. A flag indicating whether boundary padding is processed or not may be signaled in a high-level syntactic structure, such as a parameter set(s) (VPS, SPS, PPS, or APS), a slice or tile group header, or an SEI message.

[185] In the same or another embodiment, the layout information of the sub-area(s) (or sub-image(s)) may be signaled to the VPS or SPS. FIG. 17 shows an example of syntax elements in VPS and SPS. In this example, vps_sub_picture_dividing_flag is signaled to the VPS. The flag may indicate whether the input image(s) are divided into multiple subregions or not. When the value of vps_sub_picture_dividing_flag is 0, the input picture(s) in the encoded video sequence(s) corresponding to the current VPS cannot be divided into multiple sub-pictures. In this case, the size of the input image may be equal to the size of the encoded image (pic_width_in_luma_samples, pic_height_in_luma_samples) which is signaled in the SPS. When the value of vps_sub_picture_dividing_flag is 1, the input image(s) can be divided into multiple sub-pictures. In this case, the vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples syntax elements are signaled to the VPS. The values of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples can be equal to the width and height of the input image(s), respectively.

[186] In the same embodiment, the values vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples cannot be used for decoding, but can be used for layout and display.

[187] In the same embodiment, when the value of vps_sub_picture_dividing_flag is 1, the pic_offset_x and pic_offset_y syntax elements may be signaled in the SPS that correspond to a particular layer(s). In this case, the encoded image size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to a particular layer. Also, the position (pic_offset_x, pic_offset_y) of the upper left corner of the sub-region may be signaled to the SPS.

[188] In the same embodiment, position information (pic_offset_x, pic_offset_y) of the upper left corner of the sub-region cannot be used for decoding, but can be used for layout and display.

[189] In the same or another embodiment, the layout information (size and position) of all or a subset of subregions of the input image(s), dependency information between layer(s) may be signaled in a parameter set or SEI message. FIG. 18 shows an example of syntax elements for specifying layout information of sub-regions, dependencies between layers, and a relationship between a sub-region and one or more layers. In this example, the num_sub_region syntax element specifies the number of (rectangular) subregions in the current encoded video sequence; The num_layers syntax element specifies the number of layers in the current encoded video sequence. The value of num_layers can be equal to or greater than the value of num_sub_region. When any subregion is encoded as a single layer, the value of num_layers can be equal to the value of num_sub_region. When one or more subregions are encoded as multiple layers, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] indicates a dependency from the jth layer to the ith layer; num_layers_for_region[i] specifies the number of layers associated with the i-th subregion; sub_region_layer_id[i][j] specifies the layer_id of the jth layer associated with the ith subregion. The values sub_region_offset_x [i] and sub_region_offset_y [i] indicate, respectively, the horizontal and vertical position of the upper left corner of the i-th subregion. The values sub_region_width[i] and sub_region_height[i] indicate the width and height of the i-th subregion, respectively.

[190] In one embodiment, one or more syntax elements that define a set of output layers to indicate one or more layers to be output with or without profile tier level information may be signaled in a high-level syntax structure, such as VPS, DPS, SPS, PPS APS or SEI message. Referring to FIG. 19, a num_output_layer_sets syntax element indicating the number of output layer sets (OLS) in a coded video sequence related to the VPS may be signaled to the VPS. For each set of output layers, output_layer_flag can be signaled as many as there are output layers.

[191] In the same embodiment, output_layer_flag[i] equal to 1 specifies that the i-th layer is output; vps_output_layer_flag [i] equal to 0 specifies that the i-th layer is not output.

[192] In the same or another embodiment, one or more syntax elements that indicate profile tier level information for each set of output layers may be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message. Still referring to FIG. 19, a num_profile_tile_level syntax element indicating the amount of profile tier level information on the OLS in the encoded video sequence related to the VPS may be signaled to the VPS. For each set of output layers, a set of syntax elements for profile tier level information, or an index indicating specific profile tier level information among entries in the profile tier level information, may be signaled as many as there are output layers.

[193] In the same embodiment, profile_tier_level_idx[i][j] specifies the index in the profile_tier_level() syntax structure list in the VPS of the profile_tier_level() syntax structure that applies to the j-th layer of the i-th OLS.

[194] In the same or another embodiment, as shown in FIG. 20, the num_profile_tile_level and/or num_output_layer_sets syntax elements may be signaled when the number of maximum layers is greater than 1 (vps_max_layers_minus1 > 0).

[195] In the same or another embodiment, as shown in FIG. 20, a syntax element vps_output_layers_mode [i] indicating the output layer signaling mode for the i-th set of output layers may be present in the VPS.

[196] In the same embodiment, vps_output_layers_mode[i] equal to 0 indicates that only the topmost layer is output with the i-th set of output layers; vps_output_layer_mode[i] equal to 1 indicates that all layers are output with the i-th set of output layers; vps_output_layer_mode[i] equal to 2 indicates that the output layers are those with vps_output_layer_flag[i][j] equal to 1 with the i-th set of output layers. Other values may be reserved.

[197] In the same embodiment, output_layer_flag[i][j] may or may not be signaled depending on the value of vps_output_layers_mode[i] for the i-th set of output layers.

[198] In the same or another embodiment, as shown in Fig. 20, the flag vps_ptl_signal_flag [i] may be present for the i-th set of output layers. Depending on the value of vps_ptl_signal_flag[i], the profile tier level information for the i-th set of output layers may or may not be signaled.

[199] In the same or another embodiment, as shown in FIG. 21, the subpicture number, max_subpics_minus1, in the current CVS may be signaled in a high-level syntactic structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message.

[200] In the same embodiment, as shown in FIG. 21, the subpicture identifier, sub_pic_id[i], for the i-th subpicture may be signaled when the number of subpictures is greater than 1 (max_subpics_minus1 > 0).

[201] In the same or another embodiment, one or more syntax elements indicating a sub-image identifier belonging to each layer of each set of output layers may be signaled to the VPS. Referring to FIG. 22, sub_pic_id_layer[i][j][k], which indicates the k-th sub-image present in the j-th layer of the i-th set of output layers. With this information, the decoder can determine which sub-image can be decoded and output for each layer of a particular set of output layers.

[202] In an embodiment, a picture header (PH) is a syntax structure containing syntax elements that apply to all slices of the encoded picture. A picture unit (PU) is a set of NAL units that are related to each other according to a given classification rule, are sequential in decoding order, and contain exactly one encoded picture. The PU may contain a picture header (PH) and one or more NAL VCL units constituting the encoded picture.

[203] In an embodiment, the SPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId of 0, or provided through external means.

[204] In an embodiment, an SPS (RBSP) may be available to the decoding process before it is referenced and included in at least one AU with a TemporalId of 0 in a CVS that contains one or more PPSs. related to SPS, or provided through external means.

[205] In an embodiment, an SPS (RBSP) may be available to the decoding process before it is referenced by one or more PPSs, it is included in at least one PU with a nuh_layer_id equal to the smallest nuh_layer_id value of the NAL PPS units that refer to an SPS NAL unit in CVS that contains one or more SPS-related PPS or is provided through external means.

[206] In an embodiment, the SPS (RBSP) may be available to the decoding process before it is referenced by one or more PPSs, it is included in at least one PU with a TemporalId equal to 0 and nuh_layer_id equal to the smallest value nuh_layer_id of NAL PPS units that are assigned to a NAL SPS unit or provided through external means.

[207] In an embodiment, the SPS (RBSP) may be available to the decoding process before it is referenced by one or more PPSs, it is included in at least one PU with a TemporalId equal to 0 and nuh_layer_id equal to the smallest value of nuh_layer_id NAL PPS units that relate to a NAL SPS unit in a CVS that contains one or more PPS related to an SPS or is provided through external means.

[208] In the same or another embodiment, pps_seq_parameter_set_id indicates the sps_seq_parameter_set_id value for the referenced SPS. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by encoded images in CLVS.

[209] In the same or another embodiment, all SPS NAL units with a particular sps_seq_parameter_set_id value in CVS may have the same content.

[210] In the same or another embodiment, regardless of the nuh_layer_id values, the SPS NAL units may share the same sps_seq_parameter_set_id value space.

[211] In the same or another embodiment, the nuh_layer_id value of the NAL SPS unit may be equal to the smallest nuh_layer_id value of the NAL PPS units that are related to the NAL SPS unit.

[212] In an embodiment, when an SPS with nuh_layer_id equal to m is referenced by one or more PPS with nuh_layer_id equal to n , the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n , or ( direct or indirect) reference layer of a layer with nuh_layer_id equal to m .

[213] In an embodiment, the PPS (RBSP) must be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the NAL PPS unit, or provided through external means.

[214] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS that contains one or more PH (or coded slice NAL units) related to the PPS, or provided through external means.

[215] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it is included in at least one PU with nuh_layer_id equal to the smallest the nuh_layer_id value of the coded slice NAL units that refer to a PPS NAL unit in a CVS that contains one or more PHs (or coded slice NAL units) that relate to the PPS or are provided through external means.

[216] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it is included in at least one PU with a TemporalId equal to the TemporalId NAL PPS units and nuh_layer_id equal to the smallest value of the nuh_layer_id of coded slice NAL units that relate to a PPS NAL unit in a CVS that contains one or more PHs (or coded slice NAL units) that relate to a PPS or are provided through external means.

[217] In the same or another embodiment, ph_pic_parameter_set_id in PH indicates the pps_pic_parameter_set_id value for the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by encoded images in CLVS.

[218] In the same or another embodiment, all NAL PPS units with a particular pps_pic_parameter_set_id value in the PU must have the same content.

[219] In the same or another embodiment, regardless of nuh_layer_id values, NAL PPS units may share the same pps_pic_parameter_set_id value space.

[220] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the smallest nuh_layer_id value of the encoded slice NAL units that reference the PPS NAL unit.

[221] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more NAL units of an encoded slice with nuh_layer_id equal to n , the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n , or the (direct or indirect) reference layer of a layer with nuh_layer_id equal to m .

[222] In an embodiment, the PPS (RBSP) must be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the NAL PPS unit, or provided through external means.

[223] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced and included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS that contains one or more PH (or coded slice NAL units) related to the PPS, or provided through external means.

[224] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it is included in at least one PU with nuh_layer_id equal to the smallest the nuh_layer_id value of the coded slice NAL units that refer to a PPS NAL unit in a CVS that contains one or more PHs (or coded slice NAL units) that relate to the PPS or are provided through external means.

[225] In an embodiment, a PPS (RBSP) may be available to the decoding process before it is referenced by one or more PHs (or encoded slice NAL units), it is included in at least one PU with a TemporalId equal to the TemporalId NAL PPS units and nuh_layer_id equal to the smallest value of the nuh_layer_id of coded slice NAL units that relate to a PPS NAL unit in a CVS that contains one or more PHs (or coded slice NAL units) that relate to a PPS or are provided through external means.

[226] In the same or another embodiment, ph_pic_parameter_set_id in PH indicates the pps_pic_parameter_set_id value for the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by encoded images in CLVS.

[227] In the same or another embodiment, all NAL PPS units with a particular pps_pic_parameter_set_id value in the PU must have the same content.

[228] In the same or another embodiment, regardless of nuh_layer_id values, NAL PPS units may share the same pps_pic_parameter_set_id value space.

[229] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the smallest nuh_layer_id value of the encoded slice NAL units that reference the PPS NAL unit.

[230] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more NAL units of an encoded slice with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n , or the (direct or indirect) reference layer of a layer with nuh_layer_id equal to m.

[231] In the embodiment shown in FIG. 22, pps_subpic_id[i] in the image parameter set specifies the sub-image ID of the i-th sub-image. The length of the pps_subpic_id[i] syntax element is pps_subpic_id_len_minus1 + 1 bit.

[232] The variable SubpicIdVal[i] for each value of i in the range 0 to sps_num_subpics_minus1 inclusive is obtained as follows:

for( i = 0; i <= sps_num_subpics_minus1; i++ )

if( subpic_id_mapping_explicitly_signalled_flag )

SubpicIdVal[ i ] = subpic_id_mapping_in_pps_flag ? pps_subpic_id[ i ] : sps_subpic_id[ i ](80)

else

SubpicIdVal[ i ] = i

[233] In the same or another embodiment, for any two different values of i and j in the range from 0 to sps_num_subpics_minus1 inclusive, SubpicIdVal[i] may not be equal to SubpicIdVal[j].

[234] In the same or another embodiment, when the current picture is not the first CLVS picture, for each value of i in the range 0 to sps_num_subpics_minus1 inclusive, if the SubpicIdVal[i] value is not equal to the SubpicIdVal[i] value of the previous picture in decoding order in the same layer, the nal_unit_type for all NAL encoded units of a sub-image slice in the current picture with sub-image index i may be equal to a specific value in the range IDR_W_RADL to CRA_NUT, inclusive.

[235] In the same or another embodiment, when the current picture is not the first CLVS picture, for each value of i in the range 0 to sps_num_subpics_minus1 inclusive, if the SubpicIdVal[i] value is not equal to the SubpicIdVal[i] value of the previous picture in decoding order in the same layer, sps_independent_subpics_flag can be equal to 1.

[236] In the same or another embodiment, when the current picture is not the first CLVS picture, for each value of i in the range 0 to sps_num_subpics_minus1 inclusive, if the SubpicIdVal[i] value is not equal to the SubpicIdVal[i] value of the previous picture in decoding order in the same layer, subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] can be set to 1.

[237] In the same or another embodiment, when the current picture is not the first CLVS picture, for each value of i in the range 0 to sps_num_subpics_minus1 inclusive, if the SubpicIdVal[i] value is not equal to the SubpicIdVal[i] value of the previous picture in decoding order in the same layer, sps_independent_subpics_flag should be equal to 1 or subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] should be equal to 1.

[238] In the same or another embodiment, when a sub-picture is encoded independently without any reference to another sub-picture, the region sub-picture identifier value may be changed in the encoded video sequence.

[239] The number of sub-pictures in a picture may be signaled to the SPS. For example, as shown in FIG. 23, sps_num_subpics_minus1 plus 1 indicates the number of subpictures in each image in CLVS. The value of sps_num_subpics_minus1 must be in the range from 0 to Ceil(pic_width_max_in_luma_samples ÷ CtbSizeY) * Ceil(pic_height_max_in_luma_samples ÷ CtbSizeY) - 1 inclusive. If absent, the value of sps_num_subpics_minus1 is output equal to 0.

[240] In the same or another embodiment, the number of sub-pictures in a picture may be signaled in the PPS. For example, as shown in FIG. 24, pps_num_subpics_minus1 indicates the number of subpictures in each image and may be equal to sps_num_subpics_minus1.

[241] The number of slices in an image can be signaled in PPS. For example, as shown in FIG. 24, num_slices_in_pic_minus1 plus 1 indicates the number of rectangular slices in each picture related to the PPS. The value of num_slices_in_pic_minus1 can be in the range from 0 to MaxSlicesPerPicture - 1 inclusive, where MaxSlicesPerPicture is specified in Appendix A. When no_pic_partition_flag is 1, the value of num_slices_in_pic_minus1 is output as 0. When single_slice_per_subpic_flag is output as 1, the value of num_slices_in_pic_minus 1 is output equal to pps_num_subpics_minus1.

[242] The number of sub-images cannot be greater than the number of slices. When the value of single_slice_per_subpic_flag in the PPS referenced by the image is 1, the number of subpictures is equal to the number of slices in the image.

[243] In the same embodiment, the value of the variable NumSlicesInPic, which indicates the number of slices in the image, is obtained as follows: NumSlicesInPic = num_slices_in_pic_minus1 + 1.

[244] The difference value between the number of sub-pictures and the number of slices may be signaled in the PPS, and the number of slices in an image is obtained from the number of sub-pictures and the signaled value of the difference between the number of sub-pictures and the number of slices.

[245] In the embodiment shown in FIG. 25, num_slices_in_pic_minus_num_sub_pics plus pps_num_subpics_minus1 + 1 indicates the number of rectangular slices in each image related to the PPS. The value of num_slices_in_pic_minus_num_sub_pics must be in the range from 0 to MaxSlicesPerPicture - pps_num_subpics_minus1 - 1 inclusive, where MaxSlicesPerPicture is specified in Appendix A. When single_slice_per_subpic_flag is 1, then the value of num_slices_in_pic_minus_num_sub_pics is output as 0.

[246] In the same embodiment, the value of the NumSlicesInPic variable, which indicates the number of slices in the image, is obtained as follows:

if (no_pic_partition_flag == 1)

NumSlicesInPic = 1

else

NumSlicesInPic = pps_num_subpics_minus1 + num_slices_in_pic_minus_num_sub_pics + 1

[247] In the embodiment, as shown in FIG. 25, num_slices_in_pic_minus_num_sub_pics_minus1 plus pps_num_subpics_minus1 + 2 indicates the number of rectangular slices in each image related to the PPS. The value of num_slices_in_pic_minus num_sub_pics_minus1 must be in the range from 0 to MaxSlicesPerPicture - pps_num_subpics_minus1 - 2 inclusive, where MaxSlicesPerPicture is specified in Appendix A. When single_slice_per_subpic_flag is 1, then the value of num_slices_in_pic_minus_num_sub_pics_minus1 you is set to 0.

[248] In the same embodiment, the value of the NumSlicesInPic variable, which indicates the number of slices in the image, is output as follows:

if (no_pic_partition_flag == 1)

NumSlicesInPic = 1

else if ( single_slice_per_subpic_flag == 1 )

NumSlicesInPic = pps_num_subpics_minus1 + 1

else

NumSlicesInPic = pps_num_subpics_minus1 + num_slices_in_pic_minus_num_sub_pics_minus1 + 2

[249] In the same or another embodiment, when rect_slice_flag is 1, a list of NumCtusInSlice[i] for i in the range from 0 to NumSlicesInPic - 1 inclusive, indicating the number of CTUs in the i-th slice, a list of SliceTopLeftTileIdx[i] for i in range from 0 to NumSlicesInPic - 1 inclusive, indicating the tile index for the tile containing the first CTU in the slice, and the matrix CtbAddrInSlice[i][j] for i in the range from 0 to NumSlicesInPic - 1 inclusive, and j in the range from 0 to NumCtusInSlice [i] - 1 inclusive, indicating the scanning address of the image raster of the j-th CTB in the i-th slice, and the variable NumSlicesInTile [i], indicating the number of slices in the tile containing the i-th slice, are obtained as follows:

if( single_slice_per_subpic_flag ) { for( i = 0; i <= sps_num_subpics_minus1; i++ ) { NumCtusInSlice[ i ] = 0 if( subpicHeightLessThanOneTileFlag[ i ] ) /* A slice consists of a certain number of CTU rows in a tile. */AddCtbsToSlice( i, subpic_ctu_top_left_x[ i ],subpic_ctu_top_left_x[ i ] + subpic_width_minus1[ i ] + 1, subpic_ctu_top_left_y[ i ],subpic_ctu_top_left_y[ i ] + subpic_height_minus1[ i ] + 1 )else { /* A slice consists of a certain number of complete tiles covering a rectangular area. */tileX = CtbToTileColBd[ subpic_ctu_top_left_x[ i ] ]tileY = CtbToTileRowBd[ subpic_ctu_top_left_y[ i ] ]for( j = 0; j < SubpicHeightInTiles[ i ]; j++ )for( k = 0; k < SubpicWidthInTiles[ i ]; k++ ) AddCtbsToSlice( i, tileColBd[ tileX + k ], tileColBd[ tileX + k + 1 ], tileRowBd[ tileY + j ],tileRowBd[ tileY + j + 1 ] )}}} else {tileIdx = 0for( i = 0; i < NumSlicesInPic; i++ )NumCtusInSlice[ i ] = 0for( i = 0; i < NumSlicesInPic; i++ ) {SliceTopLeftTileIdx[ i ] = tileIdxtileX = tileIdx % NumTileColumnstileY = tileIdx / NumTileColumnsif( i < num_slices_in_pic_minus1 ) {sliceWidthInTiles[ i ] = slice_width_in_tiles_minus1[ i ] + 1sliceHeightInTiles[ i ] = slice_height_in_tiles_minus1[ i ] + 1} else {sliceWidthInTiles[ i ] = NumTileColumns − tileXsliceHeightInTiles[ i ] = NumTileRows − tileYNumSlicesInTile[ i ] = 1}if( slicWidthInTiles[ i ] = = 1 & & sliceHeightInTiles[ i ] = = 1 ) {(30)if( num_exp_slices_in_tile[ i ] = = 0 ) {NumSlicesInTile[ i ] = 1sliceHeightInCtus[ i ] = RowHeight[ SliceTopLeftTileIdx[ i ] / NumTileColumns ]} else {remainingHeightInCtbsY = RowHeight[ SliceTopLeftTileIdx[ i ] / NumTileColumns ]for( j = 0; j < num_exp_slices_in_tile[ i ] − 1; j++ ) {sliceHeightInCtus[ i + j ] = exp_slice_height_in_ctus_minus1[ i ][ j ] + 1remainingHeightInCtbsY −= sliceHeightInCtus[ i + j ]}uniformSliceHeight = exp_slice_height_in_ctus_minus1[ i ][ j ] + 1while( remainingHeightInCtbsY >= uniformSliceHeight ) { sliceHeightInCtus[ i + j ] = uniformSliceHeightremainingHeightInCtbsY −= uniformSliceHeightj++}if( remainingHeightInCtbsY > 0 ) { sliceHeightInCtus[ i + j ] = remainingHeightInCtbsYj++}NumSlicesInTile[ i ] = j}ctbY = tileRowBd[ tileY ]for( j = 0; j < NumSlicesInTile[ i ]; j++ ) {AddCtbsToSlice( i + j, tileColBd[ tileX ], tileColBd[ tileX + 1 ],ctbY, ctbY + sliceHeightInCtus[ i + j ] )ctbY += sliceHeightInCtus[ i + j ]}i += NumSlicesInTile[ i ] − 1} elsefor( j = 0; j < sliceHeightInTiles[ i ]; j++ )for( k = 0; k < sliceWidthInTiles[ i ]; k++ )AddCtbsToSlice( i, tileColBd[ tileX + k ], tileColBd[ tileX + k + 1 ]. = = 0 )tileIdx += ( sliceHeightInTiles[ i ] − 1 ) * NumTileColumns}}}}

[250] Where the AddCtbsToSlice(sliceIdx, startX, stopX, startY, stopY) function is specified as follows:

for( ctbY = startY; ctbY < stopY; ctbY++ )for( ctbX = startX; ctbX < stopX; ctbX++ ) {CtbAddrInSlice[ sliceIdx ][ NumCtusInSlice[ sliceIdx ] ] = ctbY * PicWidthInCtbsY + ctbX(31)NumCtusInSlice[ sliceIdx ]++ }

[251] Bitstream matching requires that the NumCtusInSlice[i] values for i in the range 0 to num_slices_in_pic_minus1 inclusive be greater than 0. Additionally, bitstream matching requires that the CtbAddrInSlice[i][j] matrix for i in the range 0 to num_slices_in_pic_minus1 inclusive and j in the range 0 to NumCtusInSlice [i] - 1 inclusive included all CTB addresses in the range 0 to PicSizeInCtbsY - 1 inclusive, once and only once.

[252] The list NumSlicesInSubpic [i] indicating the number of rectangular slices in the i-th sub-image is obtained as follows:

for( i = 0; i <= sps_num_subpics_minus1; i++ ) {NumSlicesInSubpic[ i ] = 0for( j = 0; j < NumSlicesInPic; j++ ) {posX = CtbAddrInSlice[ j ][ 0 ] % PicWidthInCtbsYposY = CtbAddrInSlice[ j ][ 0 ] / Picwidthinctbsyif ((posx> = subpic_ctu_top_left_x [i]) && (32) (posx <subpic_ctu_top_left_x [i] + subpic_width_minus1 [i] + 1) && (psy Op_left_y [i]) && (Posy <subpic_ctu_top_left_y [i] + subpic_height_minus1[ i ] + 1 ) )NumSlicesInSubpic[ i ]++}

[253] Although this disclosure describes several exemplary embodiments, there are variations, permutations, and various substitute equivalents that fall within the scope of the disclosure. Thus, it will be appreciated that those skilled in the art will be able to develop numerous systems and methods that, although not expressly shown or described herein, embody the principles of the invention and are thus within the spirit and scope thereof.

Claims (34)

1. A method for encoding video data carried out by a processor, containing: - receiving video data containing one or more sub-images; - signaling the number of sub-images and the delta value between the number of sub-images and the number of rectangular slices; And - obtaining the number of rectangular slices based on the number of sub-images and the delta value; wherein, when the current picture is not the first picture of a coded layer video sequence (CLVS), for each value of i in the range from 0 to the syntax element sps_num_subpics_minus1 inclusive, if the value of the variable SubpicIdVal[ i ] of the current picture is not equal to the value of SubpicIdVal[ i ] of the previous picture in order decoding in the same layer, the syntax element subpic_treated_as_pic_flag[ i ] and the syntax element loop_filter_across_subpic_enabled_flag[ i ] are equal to 1, and the syntax element sps_num_subpics_minus1 plus 1 indicates the number of subpictures in each picture in CLVS. 2. The method of claim 1, wherein the number of sub-images is signaled in the image parameter set. 3. The method of claim 1, wherein the number of rectangular slices is signaled in the image parameter set. 4. The method according to claim 3, wherein the number of rectangular slices is output to one based on the set flag corresponding to the image having no partitions. 5. The method of claim 1, wherein the delta value between the number of sub-images and the number of rectangular slices is signaled in the image parameter set. 6. The method of claim 5, wherein based on the set flag corresponding to a single slice for each sub-image, a delta value between the number of sub-images and the number of rectangular slices is set to zero. 7. The method according to claim 1, in which the number of sub-images is less than the number of rectangular slices. 8. Computer system for encoding video data, containing: - one or more computer-readable non-transitory storage media configured to store computer program code; And - one or more computer processors configured to access said computer program code and operate in accordance with the instructions of said computer program code, wherein said computer program code includes: - a reception code configured to cause one or more computer processors to receive video data containing one or more sub-pictures; - signaling code configured to cause one or more computer processors to signal the number of sub-pictures and the delta value between the number of sub-pictures and the number of rectangular slices; And - obtaining code configured to cause one or more computer processors to obtain the number of rectangular slices based on the number of sub-pictures and the delta value; wherein, when the current picture is not the first picture of a coded layer video sequence (CLVS), for each value of i in the range from 0 to the syntax element sps_num_subpics_minus1 inclusive, if the value of the variable SubpicIdVal[ i ] of the current picture is not equal to the value of SubpicIdVal[ i ] of the previous picture in order decoding in the same layer, the syntax element subpic_treated_as_pic_flag[ i ] and the syntax element loop_filter_across_subpic_enabled_flag[ i ] are equal to 1, and the syntax element sps_num_subpics_minus1 plus 1 indicates the number of subpictures in each picture in CLVS. 9. The computer system of claim 8, wherein the number of sub-images is signaled in the image parameter set. 10. The computer system of claim 8, wherein the number of rectangular slices is signaled in the image parameter set. 11. The computer system of claim 10, wherein the number of rectangular slices is output to one based on the flag set corresponding to the image having no partitions. 12. The computer system of claim 8, wherein the delta value between the number of sub-images and the number of rectangular slices is signaled in the image parameter set. 13. The computer system of claim 12, wherein, based on the set flag corresponding to a single slice for each sub-image, a delta value between the number of sub-images and the number of rectangular slices is set to zero. 14. The computer system according to claim 8, in which the number of sub-images is less than the number of rectangular slices. 15. A non-transitory computer-readable medium on which a computer program for encoding video data is stored, wherein the computer program is configured to cause one or more computer processors to: - receive video data containing one or more sub-images; - signal the number of sub-images and the delta value between the number of sub-images and the number of rectangular slices; And - get the number of rectangular slices based on the number of sub-images and the delta value; wherein, when the current picture is not the first picture of a coded layer video sequence (CLVS), for each value of i in the range from 0 to the syntax element sps_num_subpics_minus1 inclusive, if the value of the variable SubpicIdVal[ i ] of the current picture is not equal to the value of SubpicIdVal[ i ] of the previous picture in order decoding in the same layer, the syntax element subpic_treated_as_pic_flag[ i ] and the syntax element loop_filter_across_subpic_enabled_flag[ i ] are equal to 1, and the syntax element sps_num_subpics_minus1 plus 1 indicates the number of subpictures in each picture in CLVS. 16. The computer-readable medium of claim 15, wherein the number of sub-images is signaled in the image parameter set. 17. The computer-readable medium of claim 15, wherein the number of rectangular slices is signaled in the image parameter set. 18. The computer-readable medium of claim 17, wherein the number of rectangular slices is output to one based on the set flag corresponding to the partitionless image. 19. The computer-readable medium of claim 15, wherein the delta value between the number of sub-images and the number of rectangular slices is signaled in the image parameter set. 20. The computer-readable medium as set forth in claim 19, wherein based on the set flag corresponding to a single slice for each sub-image, a delta value between the number of sub-images and the number of rectangular slices is set to zero.