RU2808148C1 - Method and device for video coding - Google Patents

Abstract

FIELD: video encoding.

SUBSTANCE: adjustment flag is obtained for each of the at least one group of restriction information in the restriction information signalled in the encoded video bit stream. Each control flag indicates whether a group of restriction information from at least one group of restriction information corresponding to the corresponding control flag is present in the restriction information, wherein the restriction information is for a subset of encoding blocks in the encoded video bit stream. It is determined whether the control flag restriction information group is present in the restriction information, wherein the control flag restriction information group includes at least one restriction flag for a subset of encoding blocks. Prediction information for a subset of encoding blocks is determined based on whether a group of control flag constraint information is present in the constraint information. A subset of coding blocks is reconstructed based on the prediction information.

EFFECT: increased efficiency of video coding.

20 cl, 42 dwg

Description

INCLUSION BY REFERENCE

[00] This application claims priority to U.S. Patent Application No. 17/220672, "METHOD AND APPARATUS FOR VIDEO ENCODING," filed April 1, 2021, which claims priority to U.S. Provisional Application No. 63/005,345, "Method for Split Signaling" sub-image and slice in an encoded video stream,” filed April 5, 2020, US Provisional Application No. 63/010272, “Method of Signaling a Picture Header and a Slice Header in an Encoded Video Stream,” filed April 15, 2020, US Provisional Application No. 63/024140 , "Methods of Controlling Encoding Tools Using Constraint Flags in an Encoded Video Stream," filed May 13, 2020, US Provisional Application No. 63/027848, "Methods of Signaling Field Encoding Information in a Multilayer Video Stream," filed May 20, 2020, and Provisional US Application No. 63/028 222, “Methods for Conditionally Signaling Restriction Flags in an Encoded Video Stream,” filed May 21, 2020. The entire disclosure of the prior applications is incorporated herein by reference in its entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[1] The present invention describes embodiments generally related to video encoding.

BACKGROUND OF THE ART

[2] The description of the prior art presented herein is intended to provide a general context for the invention. The work of the inventors named herein, to the extent described in this prior art section, and aspects of the specification that would not otherwise be considered prior art at the time the application is filed are neither expressly nor impliedly acknowledged as prior art. prior art regarding the present invention.

[3] Video encoding and decoding can be performed using inter-frame prediction with motion compensation. Uncompressed digital video may include a series of images, with each image having a spatial size, for example, 1920x1080 luminance samples and associated chrominance samples. A series of images can have a fixed or variable image rate (informally also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has certain bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 luminance samples resolution at 60 Hz frame rate) requires a bandwidth of close to 1.5 Gbps. An hour of such video requires more than 600 GB of disk space.

[4] One of the purposes of video encoding and decoding may be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth and/or memory space requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, or a combination of both, can be used. Lossless compression refers to techniques in which an exact copy of the original signal can be reconstructed from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely used. The acceptable degree of distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio may reflect the following: Higher resolution/acceptable distortion may result in a higher compression ratio.

[5] The video encoder and video decoder may use techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[6] Video codec technologies may include techniques known as intra-coding. In intra-encoding, sample values are presented without reference to samples or other data from previously reconstructed reference images. In some video codecs, the image is spatially divided into blocks of samples. When all blocks of samples are encoded in intra-mode, this image may be an intra-image. Intra pictures and their derivatives, such as independent decoder update pictures, can be used to reset the state of the decoder and therefore can be used as the first picture in the encoded video bitstream and video session, or as a still picture. The intra-block samples may be transformed and the transform coefficients may be quantized before entropy encoding. Intra-prediction can be a method that minimizes the sample values in a region before transforming. In some cases, the smaller the post-conversion DC value and the smaller the AC coefficients, the fewer bits required for a given quantization step size to represent the block after entropy encoding.

[7] Traditional intra-coding, known for example from MPEG-2 generation encoding technologies, does not use intra-prediction. However, some new video compression technologies include methods that attempt, for example, from surrounding sample data and/or metadata obtained during encoding/decoding of spatially adjacent and preceding blocks of data in the decoding order. Such methods are henceforth called “intra-prediction” methods. It should be noted that in at least some cases, intra-prediction uses reference data only from the current image being reconstructed and not from the reference images.

[8] There can be many different forms of intra-prediction. When a given video encoding technology can use more than one of these methods, the method used may be encoded in intra-prediction mode. In some cases, modes may have sub-modes and/or parameters, and these may be individually encoded or included in the mode codeword. The choice of which codeword to use for a given mode/submode/parameter combination can have an impact on improving coding efficiency through intra-prediction, as can the entropy coding technique used to convert the codewords into a bit stream.

[9] A specific intra-prediction mode was introduced in H.264, enhanced in H.265, and further enhanced in new coding technologies such as Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). A predictor block can be generated using the values of neighboring samples belonging to already available samples. The sample values of adjacent samples are copied to the prediction block according to the direction. The reference to the direction to be used may be encoded in the bitstream or may itself be predicted.

[10] FIG. 1A depicts in the lower right corner a subset of the nine prediction directions known from the 33 possible H.265 prediction directions (corresponding to 33 angular modes out of 35 intra-modes). The point where the arrows converge (101) represents the predicted reading. The arrows indicate the direction in which the count is predicted. For example, arrow (102) indicates that sample (101) is predicted based on the sample or samples in the upper right corner at an angle of 45 degrees from the horizontal. Likewise, arrow (103) indicates that sample (101) is predicted from the sample or samples in the lower left corner of sample (101) at an angle of 22.5 degrees from the horizontal.

[11] In the upper left corner of FIG. 1A there is a square block (104) of 4x4 samples (indicated by a dashed thick line). The square block (104) includes 16 samples, each labeled "S", its position in the Y dimension (eg, row index) and its position in the X dimension (eg, column index). For example, sample S21 is the second sample in the Y dimension (top) and the first (left) sample in the X dimension. Likewise, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block size is 4x4 samples, S44 is at the bottom right. The following are reference readings, which follow a similar numbering scheme. The reference sample is labeled with the letter R, its Y position (eg row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the prediction samples are adjacent to the reconstructed block; so there is no need to use negative values.

[12] Intra-image prediction can work by copying the values of reference samples from neighboring samples according to the signal direction of the prediction. For example, assume that the coded video bitstream includes signaling that, for that block, indicates the prediction direction corresponding to arrow (102), that is, samples are predicted from the prediction sample or samples in the upper right corner at an angle of 45 degrees from the horizontal. In this case, samples S41, S32, S23 and S14 are predicted based on the same reference sample R05. Count S44 is then predicted based on reference count R08.

[13] In some cases, the values of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample; especially when the directions are not evenly divisible by 45 degrees.

[14] The number of possible destinations has increased as video coding technology has advanced. N.264 (2003) can present nine different directions. This number increased to 33 in H.265 (2013), and JEM/VVC/BMS at the time of this invention can support up to 65 directions. Experiments have been conducted to determine the most likely directions, and certain entropy encoding techniques are used to represent these probable directions in a small number of bits, taking a certain penalty for less probable directions. In addition, the directions themselves can sometimes be predicted from neighboring directions used in adjacent, already decoded blocks.

[15] FIG. 1B illustrates a diagram (180) that depicts 65 intra-prediction directions according to JEM to illustrate the increasing number of prediction directions over time.

[16] The mapping of intra-prediction direction bits in the encoded video bitstream that represent the direction may differ from video encoding technology to video encoding technology; and can range, for example, from simple direct mappings of the prediction direction to intra-prediction mode, to codewords, to complex adaptive schemes involving the most likely modes, and similar methods. However, in all cases, there may be certain directions that are statistically less likely to occur in video content than some other directions. Since the goal of video compression is to reduce redundancy, these less likely directions in a well-performing video encoding technology will be represented by a larger number of bits than the more likely directions.

[17] Motion compensation may be a lossy compression technology and may refer to technologies in which a block of sample data from a previously reconstructed image or part thereof (reference image) after being spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) is used to predict a new reconstructed image or part of an image. In some cases, the reference image may be the same as the image currently being restored. MVs may have two dimensions X and Y, or three dimensions, with the third being an indicator of the reference image used (the latter, indirectly, may be the time dimension).

[18] In some video compression techniques, a MV applicable to a particular sample data region can be predicted from other MVs, such as those belonging to another sample data region, spatially adjacent to the reconstructed region and preceding that MV in the decoding order. This can significantly reduce the amount of data required to encode MVs, thereby eliminating redundancy and increasing compression ratio. MV prediction can work effectively, for example, because when encoding the input video signal received from a camera (known as natural video), there is a statistical probability that regions larger than the region to which a particular MV applies move in a similar direction and, hence, in some cases can be predicted using a similar motion vector obtained from the MV of a neighboring region. This results in the MV found for a given region being similar or identical to the MV predicted from surrounding MVs, which in turn can be represented after entropy encoding in fewer bits than would be used in encoding MB directly. In some cases, MV prediction may be an example of lossless compression of a signal (namely, MV) obtained from a source signal (namely, a stream of samples). In other cases, the MV prediction itself may be lossy, for example, due to rounding errors when calculating a predictor from several surrounding MVs.

[19] Various MV prediction mechanisms are described in H.265/HEVC (ITU-T Rec. H.265, “High Efficiency Video Coding,” December 2016). Of the many MV prediction mechanisms that H.265 offers, a method hereafter referred to as “spatial fusion” is described.

[20] As shown in FIG. 2, the current block (201) contains samples that were detected by the encoder during the motion search process as being predicted from a previous block of the same size that was spatially shifted. Instead of encoding this MV directly, The MV may be derived from metadata associated with one or more reference pictures, for example, the most recent (in decoding order) reference picture, using the MV associated with one of five surrounding samples, designated A0, A1 and B0, B1, B2 (from 202 to 206 respectively). In H.265, MV prediction can use predictors from the same reference picture that a neighboring block uses.

DISCLOSURE OF THE INVENTION

[21] Aspects of the invention relate to methods and apparatus for video encoding/decoding. In some examples, a video decoding apparatus includes processing circuitry. The processing circuitry is configured to obtain an adjustment flag for each of the at least one group of restriction information in the restriction information signaled in the encoded video bitstream. Each control flag may indicate whether a restriction information group of at least one restriction information group corresponding to the corresponding control flag is present in the restriction information. The constraint information may relate to a subset of encoding blocks in the encoded video bitstream. The processing circuitry is configured to determine, based on a restriction information group control flag of at least one restriction information group, whether the control flag restriction information group is present in the restriction information, wherein the control flag restriction information group includes at least one restriction flag for a subset of encoding blocks. . The processing circuitry is configured to determine prediction information for a subset of encoding blocks based on whether a control flag constraint information group is present in the constraint information. The processing circuit is configured to reconstruct a subset of coding blocks based on the prediction information.

[22] In an embodiment, it is determined that a control flag restriction information group is present in the restriction information. The processing circuitry is configured to determine prediction information for a subset of encoding blocks based on at least one restriction flag in the control flag restriction information group.

[23] In an embodiment, it is determined that the control flag restriction information group is not present in the restriction information.

[24] In an embodiment, each control flag indicates that the group of restriction information corresponding to the corresponding control flag is not present in the restriction information, and there are no restriction flags in the restriction information.

[25] In an embodiment, the constraint information is present in the high-level syntax for at least one picture, and the subset of encoding blocks includes encoding blocks in at least one picture.

[26] In an embodiment, one or more of at least one group of constraint information is present in the encoded video bitstream. A plurality of restriction flags in one or more of the at least one group of restriction information are signaled in accordance with a predetermined order. The processing circuitry is configured to perform run length decoding on the plurality of constraint flags and determine prediction information for a subset of encoding blocks based on the plurality of constraint flags.

[27] In an embodiment, the at least one restriction flag in the control flag restriction information group includes a plurality of restriction flags signaled in accordance with a predetermined order. The processing circuitry is configured to perform run length decoding on a plurality of limit flags.

[28] In an embodiment, the offset and length are present in the constraint information, indicating that at least one constraint flag in the control flag constraint information group is present in the constraint information.

[29] In an embodiment, one or more of at least one group of constraint information is present in the encoded video bitstream. For each of one or more of the at least one group of constraint information, an offset and a length are present in the constraint information indicating that at least one constraint flag in the corresponding one or more of the at least one group of constraint information is present in the information restrictions. The number of offsets is signaled in the encoded video bitstream.

[30] In an embodiment, one or more of at least one group of constraint information is present in the encoded video bitstream. One or more syntax elements in the constraint information are byte aligned. The one or more syntax elements may include a control flag for each of the at least one group of restriction information and a plurality of restriction flags in one or more of the at least one group of restriction information.

[31] Aspects of the invention also relate to a non-transitory computer-readable medium storing instructions that, when executed by a computer to decode video, cause the computer to execute video decoding methods. Brief description of drawings

[32] Additional features, spirit and various advantages of the disclosed subject matter will be better understood from the following detailed description and the accompanying drawings, in which the following is shown:

[33] FIG. 1 is a schematic illustration of an exemplary subset of intra-prediction modes.

[34] FIG. 1B is an illustration of exemplary intra-prediction directions.

[35] In FIG. 2 is a schematic illustration of the current block and surrounding spatial merge candidates in one example;

[36] In FIG. 3 is a schematic illustration of a simplified block diagram of a communication system (300) in accordance with an embodiment;

[37] FIG. 4 is a schematic illustration of a simplified block diagram of a communication system (400) in accordance with an embodiment.

[38] In FIG. 5 is a schematic illustration of a simplified block diagram of a decoder in accordance with an embodiment.

[39] In FIG. 6 is a schematic illustration of a simplified block diagram of an encoder in accordance with an embodiment.

[40] FIG. 7 is a block diagram of an encoder in accordance with another embodiment.

[41] In FIG. 8 is a block diagram of a decoder according to another embodiment;

[42] In FIG. 9 shows examples of adaptive resolution change (ARC) parameter signaling according to embodiments of the present disclosure.

[43] In FIG. 10 shows examples of ARC parameter signaling according to embodiments of the disclosure.

[44] In FIG. 11 illustrates an example set of profile slice layer (PTL) syntax elements according to an embodiment of the disclosure.

[45] In FIG. 12 shows an example of general constraint information according to an embodiment of the disclosure.

[46] FIG. 13A-13B show an example of a set of PTL syntax elements according to an embodiment of the disclosure.

[47] FIG. 14A-14B show an exemplary syntax structure of general constraint information according to an embodiment of the disclosure.

[48] FIG. 15A-15D show an exemplary syntax structure of general constraint information according to an embodiment of the disclosure.

[49] FIG. 16 shows an example of a set of PTL syntax elements in a PTL syntax structure according to an embodiment of the disclosure.

[50] In FIG. 17 shows an example of a syntax table for indicating encoding information of a parameter set field in a sequence parameter set according to an embodiment of the disclosure.

[51] In FIG. 18 shows an example of a syntax table for indicating encoding information of a parameter set field in a video parameter set according to an embodiment of the disclosure.

[52] In FIG. 19 is a flowchart describing a process according to an embodiment of the invention.

[53] In FIG. 20 shows examples of syntax tables according to embodiments of the invention.

[54] FIG. 21 shows a block diagram of a picture order counter (POC) parsing and decoding cycle per access unit and a value of the access unit counter according to an embodiment of the disclosure.

[55] FIG. 22 shows an example of a prediction structure for adaptive resolution scalability according to an embodiment of the disclosure.

[56] FIG. 23 shows an example of a video bitstream structure containing multi-layer sub-images, according to an embodiment of the disclosure.

[57] FIG. 24 shows a schematic illustration of displaying a selected sub-image with improved resolution according to an embodiment of the disclosure.

[58] FIG. 25 shows a flowchart of a decoding and display process for a video bitstream containing multi-layer sub-images, according to an embodiment of the disclosure.

[59] FIG. 26 shows a schematic illustration of a 360° video display with a sub-image enhancement layer according to an embodiment of the disclosure.

[60] FIG. 27 shows an example of sub-image arrangement information and a corresponding image prediction layer and structure according to an embodiment of the disclosure.

[61] In FIG. 28 shows an example of sub-image arrangement information and a corresponding layer and image prediction structure with a local area spatial scalability modality according to an embodiment of the disclosure.

[62] FIG. 29A-29B show examples of syntax tables for sub-image arrangement information according to an embodiment of the disclosure.

[63] FIG. 30 shows an example of a SEI message syntax table for sub-image arrangement information according to an embodiment of the disclosure.

[64] In FIG. 31 shows an example of a syntax table to indicate an output layer and PTL information for a set of output layers according to an embodiment of the disclosure.

[65] In FIG. 32 shows an example of a syntax table to indicate an output layer mode for a set of output layers according to an embodiment of the disclosure.

[66] FIG. 33 shows an example of a syntax table to indicate the current sub-image of each layer for each set of output layers according to an embodiment of the disclosure.

[67] In FIG. 34 shows an example of a parameter set syntax table in a video parameter set according to an embodiment of the disclosure.

[68] In FIG. 35 shows an example of a syntax table to specify an output layer set for an output layer mode according to an embodiment of the disclosure.

[69] In FIG. 36 is a schematic illustration of a computer system in accordance with one embodiment.

IMPLEMENTATION OF THE INVENTION

[70] In FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present invention. The communications system (300) includes a plurality of terminal devices that can communicate with each other, for example, through a network (350). For example, the communication system (300) includes a first pair of terminal devices (410) and (3320) connected to each other via a network (350). In the example of FIG. 3, the first pair of terminals (3410) and (320) perform unidirectional data transfer. For example, an endpoint (310) may encode video data (eg, a stream of video images that is captured by an endpoint (310)) for transmission to another endpoint (320) via a network (350). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) may receive encoded video data from the network (350), decode the encoded video data to reconstruct video images, and display video images in accordance with the reconstructed video data. Unidirectional data transfer may be common in media serving applications and the like.

[71] In another example, the communications system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which may occur, for example, during a video conference call. For bidirectional data transmission, in an example, each endpoint of endpoints (330) and (340) may encode video data (eg, a stream of video images that is captured by the endpoint) for transmission to another endpoint of endpoints (330) and (340). ) via network (350). Each terminal device of terminal devices (330) and (340) can also receive encoded video data transmitted by another terminal device of terminal devices (330) and (340), and can decode the encoded video data to recover video images, and can display video images on an available display device according to the recovered video data.

[72] In the example of FIG. 3, end devices (310), (320), (330) and (340) may be illustrated as servers, personal computers and smartphones, but the principles of the present invention cannot be limited thereto. Embodiments of the present invention find use in laptop computers, tablet computers, media players, and/or specialized videoconferencing equipment. Network (350) is any number of networks that transmit encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and/or wireless communication networks. The communications network (350) may communicate on circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For purposes of this description, the architecture and topology of the network (350) may not be essential to the operation of the present invention unless explained below herein.

[73] FIG. 4 illustrates, as an example application for the disclosed subject matter, the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, compressed video storage on digital media including CD, DVD, memory card, etc., and so on. .

[74] The streaming system may include a capture subsystem (413), which may include a video source (401), such as a digital camera, producing, for example, a stream of video images (402) that are uncompressed. In the example, the video stream (402) includes samples captured by a digital camera. The video image stream (402), represented by a thick line to emphasize the large amount of data compared to the encoded video data (404) (or encoded video bitstreams), may be processed by an electronic device (420) that includes a video encoder (403) connected to the source (401) video. The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as a thin line to emphasize the smaller amount of data compared to the video image stream (402), may be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) in FIG. 4, may access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include a video decoder (410), for example, in an electronic device (430). A video decoder (410) decodes an incoming copy (407) of encoded video data and creates an outgoing stream of video images (411) that can be rendered on a display (412) (eg, a display screen) or other rendering device (not shown). In some streaming systems, encoded video data (404), (407) and (409) (eg, video bitstreams) may be encoded according to certain video encoding/compression standards. Examples of these standards include ITU-T Rec. H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[75] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[76] FIG. 5 shows a block diagram of a video decoder (510) according to an embodiment of the present invention. A video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (eg, receiving circuitry). A video decoder (510) may be used in place of the video decoder (410) in the example of FIG. 4.

[77] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510); in the same or another embodiment, one encoded video sequence at a time, where the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware/software connection to a storage device in which the encoded video data is stored. The receiver (531) may receive the encoded video data with other data, such as encoded audio data and/or ancillary data streams, which may be sent to their respective using entities (not shown). The receiver (531) can separate the encoded video sequence from other data. To combat network noise, a buffer memory (515) may be connected between the receiver (531) and the entropy decoder/analyzer (520) (“parser (520)”). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, it may be located outside the video decoder (510) (not shown). In other cases, there may be a buffer memory (not shown) outside the video decoder (510), for example to deal with network jitter, and further another buffer memory (515) inside the video decoder (510), for example, to handle playback time. When the receiver (531) receives data from a store/forward device with sufficient bandwidth and controllability or from an iso-synchronous network, the buffer memory (515) may not be needed or may be small. For use in best quality packet networks, such as the Internet, a buffer memory (515) may be required, which may be relatively large and may preferably be of an adaptive size, and may be at least partially implemented in the operating system or the like (not shown) outside the video decoder (510).

[78] The video decoder (510) may include a parser (520) for recovering symbols (521) from the encoded video sequence. Categories of these symbols include information used to control the operation of the video decoder (510), and potentially information for controlling a visualization device, such as a visualization device (512) (e.g., a display screen) that is not an integral part of the electronic device (530), but may be connected to an electronic device (530) as shown in FIG. 5. The control information for the rendering device(s) may be in the form of additional extended information parameter set fragments (SEI messages) or video usability information (VUI) (not shown). The parser (520) may parse/entropy decode the received encoded video sequence. The encoding of the encoded video sequence may be in accordance with a video encoding technology or standard and may follow various principles, including variable length encoding, Huffman encoding, arithmetic encoding with or without context sensitivity, and so on. The parser (520) may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), and so on. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[79] The parser (520) may perform an entropy decoding/parsing operation on the video sequence obtained from the buffer memory (515) to create symbols (521).

[80] Symbol recovery 521 may involve a variety of different modules depending on the type of encoded video image or portions thereof (such as inter and intra image, inter and intra block) and other factors. Which modules are enabled and how can be controlled by subgroup control information parsed from the encoded video sequence using the parser (520). The flow of such subgroup control information between the parser (520) and the plurality of modules is not shown below for clarity.

[81] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually divided into a number of functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with each other and can be at least partially integrated into each other. However, for purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into functional modules as follows.

[82] The first module is a scaling/inverting module (551). The scaling/inverse transform module (551) receives the quantized transform factor, as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, and so on, as symbol(s) (521) from the parser (520). The scaling/inverting module (551) may output blocks containing sample values that can be input to the aggregator (555).

[83] In some cases, the output samples of the scaling/demapping module (551) may refer to an intra-coded block; that is, to a block that does not use prediction information from previously reconstructed images, but may use prediction information from previously reconstructed portions of the current image. Such prediction information may be provided by an intra-frame prediction module (5452). In some cases, the intra-frame prediction module (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information extracted from the current image buffer (558). The current image buffer (558) buffers, for example, a partially reconstructed current image and/or a fully reconstructed current image. The aggregator (555) in some cases adds, for each sample, prediction information generated by the intra-prediction module (552) to the output sample information provided by the upscaling/inverse transform module (551).

[84] In other cases, the output samples of the scaling/demapping module (551) may be related to an intercoding and potentially motion-compensated block. In such a case, the motion compensated prediction module (553) may access the reference image memory (557) to retrieve samples used for prediction. After motion compensation of the selected samples according to the symbols (521) associated with the block, these samples can be added by the aggregator (555) to the output of the scaler/inverse module (551) (here called residual samples or residual signal) to generate information about output samples. The reference image memory addresses (557) from which the motion compensated prediction module (553) selects prediction samples may be controlled by motion vectors available to the motion compensated prediction module (553) in the form of symbols (521), which may have, for example, components X, Y and reference image. Motion compensation may also include interpolation of sample values retrieved from reference image memory (557) using precise sub-sample motion vectors, motion vector prediction mechanisms, and so on.

[85] The output samples of the aggregator (555) can be subjected to various loop filtering techniques in the loop filter module (556). Video compression technologies may include intra-loop filtering technologies that are controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and provided to the loop filter module (556) as symbols (521) from the parser (520), but may also respond to meta-information obtained during decoding of previous (in decoding order) parts of the encoded image or encoded video sequence, as well as respond to previously restored and loop-filtered sample values.

[86] The output of the loop filter module (556) may be a stream of samples, which may be output to the renderer (512) and also stored in the reference picture memory (557) for use in future inter-frame prediction.

[87] Certain encoded images, after complete reconstruction, can be used as reference images for future prediction. For example, after the encoded picture corresponding to the current picture is completely reconstructed and the encoded picture has been identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and the fresh current picture buffer may be reallocated before starting to reconstruct the next encoded picture.

[88] The video decoder (510) may perform decoding operations according to a predefined video compression technology in a standard such as ITU-T Rec. N.265. The encoded video sequence may conform to the syntax specified by the video compression technology or standard being used, in the sense that the encoded video sequence conforms to both the syntax of the video compression technology or standard and the profiles documented in the video compression technology or standard. In particular, a profile may select certain tools as the only tools available for use in that profile from all tools available in a video compression technology or standard. It may also be necessary for compliance to ensure that the complexity of the encoded video sequence is within the boundaries defined by the technology level or video compression standard. In some cases, levels limit the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the layers can in some cases be further limited by hypothetical reference decoder (HRD) specifications and HRD buffer management metadata signaled in the encoded video sequence.

[89] In an embodiment, the receiver (531) may receive additional (redundant) coded video data. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to correctly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[90] FIG. 6 is a block diagram of a video decoder (603) according to an embodiment of the present invention. A video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (eg, transmission circuitry). A video encoder (603) may be used in place of the video encoder (603) in the example of FIG. 4.

[91] The video encoder (603) may receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6), which may capture the video image(s) for encoding by the video encoder (603). . In another example, the video source (601) is part of an electronic device (620).

[92] The video source (601) may provide the original video sequence for encoding by the video encoder (603) in the form of a digital video sample stream, which may have any suitable bit depth (for example: 8 bits, 10 bits, 12 bits, ...), any color space (for example, BT.601 Y CrCb, RGB, ...) and any suitable sampling structure (for example, Y CrCb 4:2:0, Y CrCb 4:4:4). In a multimedia serving system, the video source (601) may be a storage device storing pre-prepared video. In a video conferencing system, the video source 601 may be a camera that captures local image information as a video sequence. Video data may be provided as many individual images that, when viewed sequentially, convey motion. The images themselves can be organized as a spatial array of pixels, in which each pixel can contain one or more samples depending on the sampling structure used, color space, etc. One skilled in the art can readily understand the relationship between pixels and samples. The description below is oriented towards readings.

[93] In an embodiment, the encoder (603) may encode and compress images of the original video sequence into an encoded video sequence (643) in real time or with any other timing constraints as required by the application. Providing an appropriate encoding rate is one of the functions of the controller (650). In some embodiments, the controller (650) controls other functional modules, as described below, and is operably coupled to the other functional modules. The connection is not depicted for clarity. The parameters set by the controller (650) may include parameters related to rate control (image skip, quantizer, rate-distortion optimization method lambda value, ...), image size, group of picture (GOP) layout, maximum search range motion vector and so on. The controller (650) may be configured to have other suitable functions that relate to the video encoder (603) optimized for a particular system design.

[94] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As a simplified description, in an example, the encoding loop may include a source encoder 630 (e.g., responsible for generating symbols, such as a symbol stream, based on the input image to be encoded, and the reference image(s), as well as the (local) decoder (633) built into video encoder (603).Decoder (633) reconstructs symbols to create sample data in the same way that a (remote) decoder can also create (since any compression between symbols and the encoded video bitstream is lossless in compression technologies videos discussed in the disclosed subject matter.) The reconstructed sample stream (sample data) is input into the reference picture memory 634. Since decoding the symbol stream produces bit-accurate results, regardless of the location of the decoder (locally or remotely), the contents in the memory reference pictures (634) is also bit-accurate between the local encoder and the remote encoder.In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" using prediction during decoding. This fundamental principle of reference picture synchrony (and the resulting drift if synchrony cannot be maintained, for example due to channel errors) is also used in some related fields of technology.

[95] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510), which has already been described in detail above in connection with FIG. 5. Briefly referring also to FIG. 5, however, since the symbols are available, and the encoding/decoding of the symbols into the encoded video sequence by the entropy encoder (645) and parser (520) can be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory ( 515), parser (520), cannot be fully implemented in the local decoder (633).

[96] At this point, the observation can be made that any decoding technology other than parsing/entropy decoding that is present in the decoder must also necessarily be present in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter of the invention focuses on the operation of the decoder. The description of encoding technologies may be shortened because they are the inverse of the fully described decoding technologies. Only some areas require more detailed description, which is given below.

[97] In operation, in some examples, the source encoder (630) may perform motion-compensated predictive encoding that predictively encodes the input image with reference to one or more previously encoded images from the video sequence that have been designated “reference images.” Thus, the encoding engine 632 encodes the differences between blocks of pixels in the input image and blocks of pixels in the reference image(s) that may be selected as predictive reference(s) for the input image.

[98] The local video decoder (633) may decode encoded video image data, which may be referred to as reference pictures, based on the symbols generated by the source encoder (630). The operations of the encoding engine (632) may preferably be lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence can usually be a copy of the original video sequence with some errors. The local video decoder (633) replicates the decoding processes that may be performed by the video decoder on the reference pictures and may cause the restored reference pictures to be stored in a reference picture cache (734). Thus, the video encoder 603 may locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures that will be received by the far-end video decoder (assuming no transmission errors).

[99] The predictor (635) may perform predictive searches for the encoding engine (632). That is, for a new image to be encoded, the predictor (635) may search the reference image memory (634) for sample data (as reference pixel block candidates) or certain metadata such as reference image motion vectors, block shapes, etc. further, which can serve as a suitable prediction standard for new images. The predictor 635 may operate on a block-by-pixel basis of samples to find corresponding prediction references. In some cases, as determined by the search results obtained by the predictor 635, the input image may have prediction patterns taken from a plurality of reference images stored in the reference image memory 6734.

[100] The controller (650) may control the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters used to encode video data.

[101] The output signals of all the above functional modules may be entropy encoded in an entropy encoder (645). The entropy encoder (645) translates symbols generated by various functional modules into an encoded video sequence by lossless symbol compression according to technologies such as Huffman coding, variable length coding, arithmetic coding, and so on.

[102] The transmitter (640) may buffer the encoded video sequence(s) generated by the entropy encoder (645) to prepare for transmission via a communication link (660), which may be a hardware/software connection to a storage device that will store the encoded video data. The transmitter (640) may combine encoded video data from the video encoder (630) with other data to be transmitted, such as encoded audio data and/or ancillary data streams (sources not shown).

[103] The controller (650) may control the operation of the video encoder (603). During encoding, the controller (650) may assign each encoded image a specific encoded image type, which may affect the encoding techniques that can be applied to the corresponding image. For example, images can often be categorized as one of the following image types:

[104] An intra-picture (I-picture) may be one that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow various types of intra-pictures, including, for example, Independent Decoder Refresh (“IDR”) pictures. One skilled in the art is aware of these variants of I-images and their respective applications and features.

[105] A prediction picture (P-picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most one motion vector and a reference index to predict sample values of each block.

[106] A bidirectional predicted picture (B picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most two motion vectors and reference indices to predict sample values of each block. Likewise, multi-prediction images can use more than two reference images and associated metadata to reconstruct a single block.

[107] Source images can typically be spatially divided into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. Blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the encoding assignment applied to the corresponding block images. For example, I-picture blocks may be encoded without prediction, or they may be predictively encoded with reference to already encoded blocks of the same image (spatial prediction or intra-prediction). P-picture pixel blocks may be predictively encoded, by spatial prediction, or by temporal prediction with reference to one previously encoded reference picture. B-picture blocks may be predictively encoded, by spatial prediction, or by temporal prediction with reference to one or two previously encoded reference pictures.

[108] The video encoder (603) may perform encoding operations in accordance with a predetermined video encoding technology or standard, such as ITU-T Rec. N.265. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations, that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore follow the syntax specified by the video encoding technology or standard being used.

[109] In an embodiment, the transmitter (640) may transmit additional data with encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, and so on.

[110] Video may be captured as a plurality of source images (video images) in a time sequence. Intra-frame prediction (often abbreviated as intra-prediction) uses the spatial correlation in a given image, and inter-frame prediction uses the (temporal or other) correlation between images. In the example, a particular image during encoding/decoding, called the current image, is divided into blocks. When a block in the current image is similar to a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded with a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension identifying the reference picture in the case of multiple reference pictures.

[111] In some embodiments, inter-frame prediction may use a dual prediction technique. According to the dual prediction method, two reference pictures are used, such as a first reference picture and a second reference picture, which are both preceding in the decoding order the current picture in the video (but may be in the past and future, respectively, in the display order). A block in the current picture may be encoded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block may be predicted by a combination of a first reference block and a second reference block.

[112] In addition, inter-frame prediction may use a merge mode technique to improve coding efficiency.

[113] According to some embodiments of the invention, predictions, such as inter-frame predictions and intra-frame predictions, are performed in a unit of blocks. For example, according to the HEVC standard, an image in a video sequence is divided into coding tree units (CTUs) for compression, with the CTUs in the image having the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. In general, a CTU includes three coding tree units (CTUs), which are one luminance CTU and two chroma CTUs. Each CTU can be recursively partitioned into a quadtree into one or multiple coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or 4 32x32 pixel CUs, or 16 16x16 pixel CUs. In the example, each CU is analyzed to determine the prediction type for the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) depending on temporal and/or spatial predictability. Typically, each PU includes a luminance prediction unit (LU) and two chrominance PUs. In an embodiment, the encoding prediction operation (encoding/decoding) is performed in units of prediction blocks. Using the luminance prediction block as an example of the prediction block, the prediction block includes a matrix of values (eg, luminance values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, and the like.

[114] FIG. 7 shows a diagram of a video encoder (703) according to another embodiment of the invention. The video encoder (703) is configured to receive a processing unit (eg, a prediction unit) from sample values in a current video image in a video image sequence and encoding the processing unit into a coded image that is part of the encoded video sequence. In one example, a video encoder (703) is used instead of the video encoder (403) in the example of FIG. 4.

[115] In the HEVC example, the video encoder 703 receives a matrix of sample values for a processing block, such as an 8x8 sample prediction block and the like. The video encoder (703) determines whether the processing block is best encoded using intra-mode, inter-mode, or a dual-prediction mode using, for example, rate-distortion optimization. When a processing block is to be encoded in intra-mode, the video encoder (703) may use an intra-prediction method to encode the processing block into a coded image; and when a processing block is to be encoded in inter-mode or dual-prediction mode, the video encoder 703 may use an inter-prediction or dual-prediction method, respectively, to encode the processing block into a coded image. In some video encoding technologies, the fusion mode may be a sub-mode of inter-frame prediction, in which a motion vector is derived from one or more motion vector predictors without taking advantage of the encoded motion vector component outside the predictors. Some other video encoding technologies may have a motion vector component applicable to the block in question. In one example, the video encoder (703) includes other components, such as a mode selection module (not shown), for determining the mode of the processing blocks.

[116] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a common controller ( 721) and an entropy encoder (725) connected together as shown in FIG. 7.

[117] The inter-encoder (730) is configured to receive samples of a current block (eg, a processing block), compare the block to one or more reference blocks in reference pictures (eg, blocks in previous pictures and subsequent pictures), generate inter-prediction information (eg, describing redundant information according to an inter-coding method, motion vectors, merge mode information) and calculating inter-prediction results (eg, predicted block) based on the inter-prediction information using any suitable method. In some examples, the reference pictures are decoded reference pictures that are decoded based on the encoded video information.

[118] The intra encoder (722) is configured to receive samples of the current block (eg, a processing block), in some cases compare the block with blocks already encoded in the same picture, generate quantized coefficients after transformation, and in some cases also intra-encoder information - predictions (eg, intra-prediction direction information according to one or more intra-encoding methods). In one example, the intra-encoder (722) also calculates intra-prediction results (eg, a predicted block) based on the intra-prediction information and reference blocks in the same image.

[119] The common controller (721) is configured to determine common control data and control other components of the video encoder (703) based on the common control data. In one example, a common controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra-mode, the general controller (721) controls the switch (726) to select the intra-mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra-prediction information and enable intra-prediction information into the bitstream; and when the mode is inter-mode, the general controller (721) controls the switch (726) to select the inter-prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select the inter-prediction information and enable the information inter-predictions into the bitstream.

[120] A residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction results selected from the intra-encoder (722) or inter-encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In one example, a residual encoder (724) is configured to convert residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then subjected to quantization processing to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a remainder decoder (728). The remainder decoder (728) is configured to perform deconversion and generate decoded remainder data. The decoded remainder data may be suitably used by the intra-encoder (722) and inter-encoder (730). For example, the inter-encoder (730) may generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (722) may generate decoded blocks based on the decoded residual data and intra-prediction information. The decoded blocks are processed accordingly to generate decoded pictures, and the decoded pictures may be buffered in a memory circuit (not shown) and used as reference pictures in some examples.

[121] An entropy encoder (725) is configured to format the bitstream to include an encoded block. The entropy encoder (725) is configured to include various information in accordance with a suitable standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (eg, intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. It should be noted that according to the disclosed aspect of the invention, when encoding a block in the merge sub-mode of either the inter-mode or the dual-prediction mode, there is no residual information.

[122] FIG. 8 shows a circuit diagram of a video decoder (9810) according to another embodiment of the invention. The video decoder (810) is configured to receive encoded images that are part of an encoded video sequence and decode the encoded images to create reconstructed images. In one example, a video decoder (810) is used instead of the video decoder (410) in the example of FIG. 4.

[123] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an interdecoder (880), a remainder decoder (873), a recovery module (874), and an intradecoder (872) connected together as shown in Fig.8.

[124] The entropy decoder (871) may be configured to recover from the encoded image certain symbols that represent the syntax elements that make up the encoded image. Such symbols may include, for example, the mode in which the block is encoded (such as, for example, intra-mode, inter-mode, dual-prediction mode, the latter two in a merge sub-mode or other sub-mode), prediction information (such as, e.g., intra-prediction information or inter-prediction information) that may identify a specific sample or metadata that is used to predict by the intra-decoder (872) or inter-decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients and etc. In one example, when the prediction mode is inter-mode or bidirectional prediction mode, inter-prediction information is provided to the inter-decoder (880); and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may be inversely quantized and provided to a residual decoder (873).

[125] An inter-decoder (880) is configured to receive inter-prediction information and generate inter-prediction results based on the inter-prediction information.

[126] 8 An intra decoder (772) is configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[127] The residual decoder 873 is configured to perform inverse quantization to extract the dequantized transform coefficients, and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The remainder decoder (873) may also require certain control information (to enable the quantizer parameter (QP)), and this information may be provided by the entropy decoder (871) (the data path is not shown since this may only be low-volume control information).

[128] The reconstruction module (874) is configured to combine in the spatial domain the residual output by the residual decoder (873) and the prediction results (output by the inter- or intra-prediction modules, as appropriate) to form a reconstructed block, which may be part of recovered image, which in turn can be part of the recovered video. It should be noted that other suitable operations such as a release operation and the like may be performed to improve the visual quality.

[129] It should be noted that video encoders (403), (603) and (703) and video decoders (410), (510) and (810) may be implemented using any suitable method. In an embodiment, video encoders (403), (603) and (703) and video decoders (410), (510) and (810) may be implemented using one or more integrated circuits. In another embodiment, video encoders (403), (603) and (603) and video decoders (410), (510) and (810) may be implemented using one or more processors that execute software instructions.

[130] Aspects of the disclosure include controlling encoding tool(s) and functionality via restriction flags on the encoded video stream.

[131] Video encoders and video decoders can operate with a given picture size that is defined and remains constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2 it is known that the system design changes the horizontal resolution (and thus the image size) depending on factors such as scene activity, but only for I-pictures, hence typically for GOP. Resampling reference images to use different resolutions in CVS is known, for example, from ITU-T Recommendation H.263 Annex P. However, the image size does not change, only the reference images are resampled, as a result of which only parts of the image canvas can be used (for example, with downsampling ) or capture only part of the scene (for example, upsampling). Some examples, such as H.263 Annex Q, allow an individual macroblock to be oversampled by a factor of two in each dimension (eg, up or down). The image size may remain the same. The macroblock size may be fixed, such as in H.263, and therefore does not need to be signaled.

[132] The image size of the predicted images may vary. An example such as VP9 allows resampling of the reference image and resampling of the entire image. For example, (including, for example, Hendry, et al., "On Adaptive Resolution Changes (ARC) for VVC," Joint Video Team JVET-M0135-v1, January 9-19, 2019, the contents of which are incorporated herein in their entirety ) it is possible to resample the entire reference image at a different resolution (for example, higher resolution or lower resolution). The different resolution options can be encoded in a Sequence Parameter Set (SPS) and can be referenced using syntax elements for each image in a Picture Parameter Set (PPS).

[133] The present disclosure discloses methods for signaling an adaptive image size in a video bitstream.

[134] When images are encoded into a bitstream that includes one or more layers of different qualities, the bitstream may have syntax elements that specify which layer (or layers) (or set of layers) can be output at the decoder side. A set of output layers can be defined as a set of output layers. In an example video codec that supports multiple layers and scalability, one or more sets of output layers may be signaled in a video parameter set (VPS). Syntax elements defining a profile tier level (PTL) for an entire bitstream or one or more sets of output layers may be signaled in a VPS, DPS, which is identical to decoder capability information (DCI), SPS, PPS, SEI message, or the like. The PTL information may contain general restriction information, which may indicate restrictions on coding tools or functionality. It is desirable to efficiently represent and signal constraint information for various encoding tools and functions.

[135] A sub-image may refer, for example, to a rectangular arrangement of samples, blocks, macroblocks, encoding units, or similar objects that are semantically grouped and can be independently encoded at a different resolution. One or more sub-images may form an image. One or more coded sub-pictures may form a coded picture. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In some examples, one or more encoded image fragments may be collected in a compressed region without being recoded to a sample level into the encoded image. In some examples, one or more encoded image fragments may be extracted from the encoded image in the compressed domain.

[136] Adaptive resolution resampling (ARC) can refer to mechanisms that allow the resolution of an image or sub-image to be changed in CVS, for example by resampling a reference image. The ARC parameters may refer to control information used to perform adaptive resolution changes. The ARC parameters may include filter parameters, scaling factors, output and/or reference image resolutions, various control flags, and/or the like.

[137] The above description can be applied to encoding and decoding of a single and semantically independent encoded video image. Before describing the encoding/decoding of multiple sub-pictures with independent ARC parameters and the additional complexity implied, the ARC signaling parameters are described.

[138] FIG. 9 shows examples (eg, embodiments) for signaling ARC parameters according to embodiments of the disclosure. Coding efficiency, complexity, and architecture may vary between examples. A video coding standard or technology may select one or more examples or other options for signaling ARC parameters. The examples may not be mutually exclusive, and may be interchanged depending on application needs, standard technologies, encoder choice, and/or the like. [139] ARC parameter classes may include:

• Upsampling and/or downsampling factors, individual or combined in the X and Y dimensions, where one or more short syntax elements may be encoded that may indicate a table including the upsampling and/or downsampling factors;

• Upsampling and/or downsampling factors with the addition of a time dimension indicating a constant rate of increase and/or reduction for a given number of images, where one or more short syntax elements may indicate a table including the upsampling and/or downsampling factors with the addition of time measurements can be coded;

• is the resolution in the X or Y dimension, in units of samples, blocks, macroblocks, CUs, or any other suitable granularity of the input image, output image, reference image, coded image, in combination or alone. If there is more than one resolution (eg, one resolution for the input image, another resolution for the reference image), a set of values can be derived from another set of values, which can be adjusted, for example, using flags, as described below;

• Deformation coordinates similar to those used in Annex R H.263, with a suitable level of detail as described above. Annex R H.263 defines an efficient way to encode deformation coordinates. Other effective methods may be developed. For example, reverse Huffman-style variable-length encoding for Annex P warp coordinates can be replaced by suitable-length binary encoding, where the length of the binary codeword can be derived from the maximum image size multiplied by a factor and shifted by a value to account for the warp per outside the maximum image size limits;

• Upsampling and/or downsampling filter options. For example, there is only one filter for upsampling and/or downsampling. For example, it is preferable to provide greater flexibility in filter design. The example signals the filter parameters. Filter options can be selected using an index in the list of possible filter designs. The filter may be fully specified (e.g., by defining a list of filter coefficients, using suitable entropy encoding techniques), the filter may be implicitly selected through upsampling or downsampling coefficients that are signaled according to any of the mechanisms described above, and/or the like, the like, similar.

[140] The present description can be applied to encoding a finite set of upsampling or downsampling coefficients (the same coefficient is used in both the X and Y dimensions) specified by a codeword. The codeword may preferably be of variable length, encoded for example using Ext-Golomb code for certain syntax elements in video coding specifications (eg, H.264 and H.265). A suitable mapping of values for upsampling or downsampling factors may, for example, be based on Table 1.

[141] Similar mappings can be developed in accordance with the application and capabilities of the upscaling and downscaling mechanisms available in the video compression technology or standard. Table 1 can be expanded to additional values. Values can be represented by entropy encoding mechanisms other than Exponential Golomb codes, such as using binary encoding. In one example, entropy encoding mechanisms other than Ext-Golomb code may have certain advantages when oversampling coefficients are of interest outside of video processing subsystems (eg, encoder and decoder), such as multimedia-aware network elements (MANEs). When resolution change is not required (e.g. source/target resolution corresponds to 1 in Table 1), a short Ext-Golomb code can be selected (e.g. only one bit shown in Table 1), which may have an advantage in terms of encoding efficiency, e.g. , compared to using binary codes for the most common case.

[142] The number of entries in Table 1 and the associated semantics can be fully or partially customized. For example, the basic circuit of Table 1 is signaled in a high-level parameter set such as SPS or DPS. Alternatively or in addition, one or more tables like Table 1 may be defined in a video coding technology or standard and may be selected, for example, using SPS or DPS.

[143] The above is a description of how the upsampling or downsampling factor (ARC information) encoded as described above can be included in a video encoding technology or standard syntax. Similar observations can be applied to one or more codewords driving upsampling or downsampling filters. A description of the case where a filter or other data structures require a relatively large amount of data is given below.

[144] Referring to FIG. 9, in an example such as the H.263 application P, ARC information (902) in the form of four warp coordinates is included in the image header (901), in particular, in the PLUSPTYPE header extension (903) H. 263. A picture header (901) including ARC information (902) and an H.263 PLUSPTYPE header extension (903) may be used when i) a picture header is available and ii) frequent changes to the ARC information are expected. However, the overhead when using H.263 style signaling may be high, and scaling factors may not apply to image boundaries since the image header may be temporary.

[145] Referring to FIG. 9, in an example such as JVCET-M135-v1, the ARC reference information (905) (eg, index) located in the PPS (904) may point to a table (or target resolution table) ( 906) including target permissions (eg, permissions 1-3), with the table (906) located within the SPS (907). Placing target permissions in a table (906) in the SPS (907) can be justified by using the SPS as a point of interoperability negotiation during capability exchange. The resolution may vary within the limits set by the values (eg, resolutions 1-3) in the table (906) from one image to another by reference to the corresponding PPS (904).

[146] Returning to FIG. 9, additional options or examples may be used to convey ARC information in the video bitstream. Multiple options can be present simultaneously in the same video encoding technology or standard.

[147] In an embodiment, ARC information (909), such as a resampling factor (or scale factor), may be present in a header, such as a slice header, a GOB header, a tile header, a tile group header, and the like. Header (908) tile group is shown in Fig. 9 as an example. An embodiment may be adequate when the memory space for the ARC information 909 is small, for example, for a single variable length ue(v) or a fixed length codeword of several bits, as shown above.

[148] Having ARC information (909) in a header (e.g., a tile group (908) header in FIG. 9, a slice header, or a tile header) directly may have the additional benefit that the ARC information (909) can be applied to a sub-image , represented, for example, by the corresponding group of tiles (or slice, tile), and not by the entire image. Additionally, as an example, even if the video compression technology or standard only provides for changing the adaptive resolution of the entire image (as opposed to, for example, adaptive resolution changes on a per-tile group basis), placing ARC information (909) in the tile group header (908) Instead of placing ARC information (909) in the H.263 style image header, it may have certain advantages in terms of error tolerance.

[149] Referring to FIG. 9, ARC information 912 may be present in a parameter set 911, such as a PPS, a header parameter set, a tail parameter set, an adaptation parameter set (APS), and the like. APS (911) is shown in Fig. 9 as an example. The size of the parameter set (911) may preferably be no larger than an image, for example a group of tiles. The use of ARC information (eg, ARC information (912)) can be made implicit by activating an appropriate set of parameters (eg, APS (911)). For example, when a video coding technology or standard considers only image-based ARC, PPS or equivalent may be appropriate.

[150] In some embodiments, the ARC reference information (913) may be present in a tile group header (914) or similar data structure (eg, image header, slice header, tile header, or GOP header) as described above. The tile group header (914) is shown in FIG. 9 as an example. ARC reference information (913) may refer to a subset of ARC information (915) available in a parameter set (916) with a scope beyond a single image, such as SPS, DPS, etc. SPS (916) is shown in FIG. .9 as an example.

[151] FIG. 10 shows examples of ARC parameter signaling according to embodiments of the disclosure. FIG. 10 shows examples of syntax diagrams used in video coding standards. In the example, the syntax diagram notation roughly corresponds to C-style programming. Lines in bold may indicate syntax elements present in the bitstream, and lines without bold may indicate control flow(s) or setting variables.

[152] As shown in FIG. 10, a tile group header (1001), as an exemplary syntactic structure of a header applicable to a portion (eg, rectangular) of an image, may conventionally include an exponential Golomb code encoded variable length syntax element dec_pic_size_idx (1002 ) (in bold). The presence of a syntax element (eg, dec_pic_size_idx (1002)) in the tile group header (1001) can be adjusted based on an adaptive resolution, such as represented by a flag (eg, adaptive_pic_resolution_change_flag) (1003). The value of the flag (eg adaptive_pic_resolution_change_flag) (1003) is not in bold, and thus the flag is present in the bitstream at the point where the flag occurs in the syntax diagram. Information about whether an adaptive resolution is used for an image or a portion thereof may be signaled in a high-level syntactic structure (eg, SPS (1010), as described below) within or outside the bitstream.

[153] FIG. 10 shows an excerpt from SPS (1010). The first syntax element shown is a flag (for example, adaptive_pic_resolution_change_flag) (1011). When true, the flag may indicate the use of adaptive permission, which may require certain control information. For example, certain control information is present conditionally based on the value of the flag (1011), as indicated by the if() statement (1012) in the SPS (1010) and the tile group header (1001).

[154] When an adaptive resolution is used, such as shown in the example of FIG. 10, the output resolution may be encoded in sample units (or output image resolution) (1013). Output resolution (1013) can refer to both resolution width (eg output_pic_width_in_luma_samples) and resolution height (eg output_pic_height_in_luma_samples). The width and height of the resolution can determine the output resolution (1013). The video encoding technology or standard may define certain limitations on the output resolution value(s) (1013). For example, the level definition may limit the number of total output samples (for example, the product of output_pic_width_in_luma_samples and output_pic_height_in_luma_samples). A video encoding technology or standard, or an external technology or standard (e.g., a system standard) may limit the range (e.g., resolution width/resolution height is divided by a power of 2), aspect ratio (e.g., resolution width to resolution height ratio is 4:3 or 16 :9) or the like. In the example, the above restrictions may be introduced to facilitate hardware implementations.

[155] In some applications, the encoder may instruct the decoder to use a certain size of the reference image, rather than implicitly assuming that the size is the size of the output image. For example, a syntax element (eg, reference_pic_size_present_flag) (1014) controls the conditional presence of reference picture sizes (1015). The reference picture dimensions (1015) may refer to both width (eg, reference_pic_width_in_luma_samples) and height (eg, reference_pic_height_in_luma_samples).

[156] FIG. 10 shows a table of applicable decoding image width and height values. A table may be expressed by specifying a table (eg, by the syntax element num_dec_pic_size_in_luma_samples_minusl) (1016). "minus1" may refer to the interpretation of the meaning of the syntax element (1016). For example, if the encoded value is zero, there is one entry in the table. If the coded value is five, there are six entries in the table. For each row in the table, the width and height of the decoded image are included in the syntax elements (1017).

[157] Table entries represented by syntax elements (1017) can be indexed using the dec_pic_size_idx syntax element (1002) in the header of a tile group (1001), allowing different decoded sizes and scaling factors to be used for each tile group.

[158] Some video coding technologies or standards (eg, VP9) can support spatial scalability by implementing certain forms of reference image resampling in combination with temporal scalability and thus can provide spatial scalability. In an embodiment, the reference image is upsampled using ARC-style techniques to a higher resolution to form the base of the spatial enhancement layer. The upsampled image can be refined using conventional prediction mechanisms (eg, motion-compensated prediction for inter-prediction from reference images) at high resolution, for example, to add detail.

[159] The disclosure may be used in such an environment. In some examples, a value in the network abstraction layer (NAL) block header, such as a temporary identifier field, is used to indicate the temporal and spatial layers. The benefits may be that existing Selected Forwarding Units (SFUs) can be used without modification for scalable environments. SFUs can be created and optimized for forwarding selected at the temporal layer based on the value of the NAL block header's temporary identifier. To make this possible, a mapping may be performed between the encoded image size and the temporal layer, as indicated by the temporary identifier field in the NAL block header.

[160] When images are encoded into a bitstream that includes one or more layers of different qualities, the bitstream may have syntax elements that specify which layer (or layers) can be output at the decoder side. A set of output layers can be defined as a set of output layers. In a video codec that supports multiple layers and scalability, one or more sets of output layers may be signaled to one or more VPSs. One or more syntax elements that define PTL information for the entire bitstream, CVS, each set of output layers, and/or the like may be signaled in a high-level syntax structure such as VPS, DPS, DCI, SPS, PPS, APS, GOP , sequence, header, SEI message, etc.

[161] Block-level coding tools can be used to decode pixels or samples in an image to reconstruct the image. Block-level coding tools may include any suitable coding tools used in reconstructing a coding block, such as inter-prediction coding tools (or inter-coding tools), intra-prediction coding tools (or intra-coding tools), adaptive contour filter (ALF), deblocking filter (DBF), entropy coding, transform, etc.

[162] High Level Syntax (HLS) may define information about functionality, system interface, image level controls, buffer management, and the like. For example, HLS can specify a partition (e.g. tile, slice, sub-image), buffer management, random access (e.g. IDR, pure random access (CRA)), parameter set(s) (e.g. VPS, SPS, PPS, APS) , reference picture resampling (RPR), scalability and/or the like. High level syntax can be higher than block level.

[163] The control information may include SPS layer tool control information, PPS layer tool control information, sequence layer control information, bitstream layer control information and/or the like.

[164] Restriction flags may be part of the HLS structure.

[165] In the example, the restriction flags indicate tool control. The restriction flags may be provided in one of: sequence level control information and bitstream level control information. In one example, if certain tools are disabled by restriction flags, control information is not presented in the HLS and the tools are not used, for example, for coding blocks in the region corresponding to the HLS.

[166] Constraint flags may be signaled within the profile syntax structure.

[167] The restriction information may indicate whether particular tools, functionality, or syntax elements are present or used in the corresponding video stream or encoded video bitstream. For example, restriction flags may indicate whether intercoding tools, intracoding tools, DBF, entropy encoding, transform, partitioning (e.g., tile, slice), buffer management, random access (e.g., IDR), parameter set (e.g., SPS) can , PPS) and/or the like. present or used in the encoded video bitstream. The restriction information may be signaled in parameter sets (eg SPS, VPS or DCI). Constraint flags may be signaled in a high-level syntactic structure (eg, SPS, VPS, DCI).

[168] FIG. 11 shows an example of a set of PTL syntax elements according to an embodiment of the disclosure. The set of PTL syntax elements may include general_profile_idc, general_tier_flag, general_level_idc, num_sub_profiles, general_sub_profile_idc, sublayer_level_present_flag, ptl_alignment_zero_bit, sublayer_level_idc, and general constraint information.

[169] The general restriction information may include restriction information for video source type, encoding tools, functionality and/or the like. FIG. 12 shows an example of general constraint information according to an embodiment of the disclosure. The general restriction information may include a plurality of restriction flags, as shown in FIG. 12. Referring to FIG. 12, a constraint flag (eg, intra_only_constraint_flag) (1205) of 1 indicates that sh_slice_type is I (ie, the slice is an intra-slice). A constraint flag (eg, intra_only_constraint_flag) (1205) equal to 0 imposes no constraint (eg, sh_slice_type equal to I) on all coded pictures within the PTL information, where other information (eg, profile_idc) may allow non-intra slices. In an example, a constraint flag (e.g., no_alf_constraint_flag) (1206) of 1 may indicate that sps_a_lf_enabled_flag is 0 for all CVS within the scope of the PTL information, and thus adaptive loop filtering is not used even if adaptive loop filtering is enabled filtering based on, for example, profile_idc. A constraint flag (eg no_alf_constraint_flag) (1206) set to 0 does not impose the above constraint.

[170] A constraint flag (eg, no_lossless_coding_tool_constraint-flag) (1201) may be signaled in general constraint information, as shown in FIG. 12. A constraint flag (eg, no_lossless_coding_tool_constraint_flag) (1201) equal to 1 may indicate that coding tools related to lossless coding cannot be used within the scope of the PTL information, including the constraint flag (1201). A constraint flag (eg no_lossless_coding_tool_constraint_flag) (1201) of 0 does not impose the above constraint.

[171] A constraint flag (1202) (eg, no_lossy_coding_tool_constraint_flag) may be signaled in general constraint information, as shown in FIG. 12. A constraint flag (eg, no_lossy_coding_tool_constraint_flag) (1202) equal to 1 may indicate that the coding tool(s) related to lossy coding cannot be used within the scope of the PTL information including the constraint flag (1202). A constraint flag (eg no_lossy_coding_tool_constraint_flag) (1202) of 0 does not impose the above constraint.

[172] In an embodiment, the constraint flag (eg, no_lossless_coding_tool_constraint_flag) (1201) may not be equal to 1 when the constraint flag (eg, no_lossy_coding_tool_constraint_flag) (1202) is 1. Alternatively, the constraint flag (eg, no_lossy_coding_tool_constraint_flag) (1202) may not be equal to 1 when the constraint flag (for example, no_lossless_coding_tool_constraint_flag) (1201) is equal to 1.

[173] A plurality of restriction flags in the common restriction information may be sorted in a specific order. The order may be established, for example, on the basis of the likelihood that the relevant machinery and/or tools will not be used to the extent of the PTL. The order can be called priority order. The order can be represented in the general structure of the constraint information syntax from high priority to low priority, where high priority indicates that non-use of the tool (or mechanism) has a high probability, and low priority indicates that non-use of the tool (or mechanism) has low probability. Additional factors affecting the order may include tools that are likely to be used only for specific use cases (e.g. sub-image tools, scalability and/or interleaving support), the impact of the tool on encoder/decoder/implementation complexity, etc. P.

[174] FIG. 13A-13B show an example of PTL information (including, for example, a set of PTL syntax elements) according to an embodiment of the disclosure. A syntax element indicating the number of constraint flags (eg, num_available_constraint_flags) (1301) may be signaled in the PTL information. For example, the number of limitation flags may be signaled outside of the general limitation information, as shown in FIG. 13A. Alternatively, a syntax element indicating the number of constraint flags may be signaled at the beginning of the general constraint information syntax structure. When a syntax element (eg, num_available_constraint_flags) is present and the value of the syntax element (eg, num_available_constraint_flags) is N, the first N constraint flags may be present in the general constraint information syntax structure. Additionally, other constraint flags may be missing and can be inferred to be equal to a specific value. N can be a non-negative integer.

[175] In an embodiment, the value of N (eg, num_available_constraint_flags) ranges from 0 to the maximum number of constraint flags (eg, MaxNumConstraintFlags). The maximum number of limit flags can be any positive integer. The value of the maximum number of constraint flags (eg, MaxNumConstraintFlags) may be predetermined to be 16, 32, 64, 128, or the like. When the value of N (for example, num_available_constraint_flags) is 0, there are no constraint flags in the general constraint information syntax structure. The encoding of the N value (eg, num_available_constraint_flags) can be chosen such that the corresponding entropy encoded representation for the N value and constraint flags can be added to a multiple of 8 to provide byte alignment.

[176] According to aspects of the disclosure, restriction flags may be classified into one or more restriction information groups. Each group of restriction information may include one or more restriction flags and may have a corresponding throttling flag. The control flag of the corresponding restriction information group may indicate whether the restriction flag(s) may be present in the corresponding restriction information group. In one example, the throttling flag may be referred to as a restriction group presence flag. In general, a control flag is associated with a corresponding group of restriction information and is associated with the restriction flag(s) in the corresponding group of restriction information. In an embodiment, the control flag determines whether the restriction flag(s) are present (or signaled) in the restriction information in the corresponding group of restriction information. For example, if the control flag of the corresponding restriction information group is 1, the restriction flag(s) corresponding to the restriction information group may/may be present in the restriction information, for example. If the control flag of the corresponding restriction information group is 0, the restriction flag(s) corresponding to the restriction information group may/may not be present in the restriction information, for example. In the example, if all throttling flags are 0, there are no limiting flags.

[177] Restriction flags can have different scopes. For example, the domain of the restriction flags in DCI may be a coded video bitstream. Limit flags in a VPS can be CLVS with multiple layers. The restriction flag area in the SPS can be one CLVS.

[178] A flag (eg, gate flag all) may indicate whether throttling flag(s) are present. In one example, the flag is false, indicating that there are no throttling flags. In the example where there are no control flag(s), there are no restriction flags in the restriction information. In the example, the flag is true, indicating the presence of one or more throttling flags.

[179] According to aspects of the disclosure, a control flag for each of the at least one group of restriction information in the restriction information signaled in the encoded video bitstream may be obtained. Each control flag may indicate whether a restriction information group of at least one restriction information group corresponding to the corresponding control flag is present in the restriction information. The constraint information may relate to a subset of encoding blocks in the encoded video bitstream. Whether the restriction information group of the control flag is present in the restriction information can be determined based on the control flag of the restriction information group of the at least one restriction information group. The control flag restriction information group may include at least one restriction flag for a subset of coding blocks. Prediction information for a subset of encoding blocks can be determined based on whether a control flag constraint information group is present in the constraint information, the subset of encoding blocks can be reconstructed based on the prediction information.

[180] In an embodiment, it is determined that a control flag restriction information group is present in the restriction information. The prediction information for a subset of coding blocks may be determined based on at least one restriction flag in the control flag restriction information group. In an embodiment, it is determined that the control flag restriction information group is not present in the restriction information.

[181] In an embodiment, each control flag indicates that a group of restriction information corresponding to the corresponding control flag is not present in the restriction information, and there are no restriction flags in the restriction information.

[182] In an embodiment, the constraint information is present in the high-level syntax for at least one picture, and the subset of encoding blocks includes encoding blocks in at least one picture. High level syntax can be VPS, DPS, DCI, SPS, PPS, APS, GOP, sequence, etc. In the example, the high level syntax is DCI or DPS, the amount of restriction information includes the coded video bitstream and thus includes encoding blocks in the coded video bitstream. In the example, the high level syntax is VPS, the scope of the constraint information includes CLVS with multiple layers and thus includes encoding blocks in CLVS. In the example, the high level syntax is SPS, the scope of the constraint information includes one CLVS and thus includes encoding blocks in one CLVS.

[183] In an embodiment, each restriction flag that may be present (or signaled) in the restriction information is classified into one or more restriction information groups and thus can be adjusted by a corresponding adjustment flag. In an example, the one or more groups of restriction information include a first group of restriction information and a second group of restriction information. The restriction flags include at least one first restriction flag and at least one second restriction flag. The at least one first restriction flag is classified into a first restriction information group, and the at least one second restriction flag is classified into a second restriction information group. The first control flag of the first constraint information group may indicate whether at least one first constraint flag (or first constraint information group) is present (eg, signaled) in the constraint information. The second control flag of the second group of restriction information may indicate whether at least one second restriction flag (or second group of restriction information) is present (eg, signaled) in the restriction information. Thus, in the example, the restriction information does not include additional restriction flag(s) for the at least one first restriction flag and the at least one second restriction flag.

[184] In an embodiment, the first subset of constraint flags is classified into one or more constraint information groups and thus can be adjusted by a corresponding throttling flag. In addition, the second subset of restriction flags is outside of one or more groups of restriction information and thus is not controlled by the throttling flag(s). In an example, the one or more groups of restriction information include a third group of restriction information and a fourth group of restriction information. The first subset of restriction flags includes at least one third restriction flag and at least one fourth restriction flag. The at least one third restriction flag is classified into a third restriction information group, and the at least one fourth restriction flag is classified into a fourth restriction information group. The third control flag of the third group of restriction information may indicate whether at least one third restriction flag (or third group of restriction information) is present (eg, signaled) in the restriction information. The fourth control flag of the fourth group of restriction information may indicate whether at least one fourth restriction flag (or fourth group of restriction information) is present (eg, signaled) in the restriction information. In addition, the restriction information may include a second subset of restriction flags. In the example, the third control flag and the fourth control flag indicate that the third group of restriction information and the fourth group of restriction information are not signaled in the restriction information, and thus the restriction information includes the second subset of restriction flags and does not include at least one third restriction flag and at least one fourth restriction flag. In the example, the third control flag and the fourth control flag indicate that the third group of restriction information and the fourth group of restriction information are reported in the restriction information, and thus the restriction information includes a second subset of restriction flags, at least one third restriction flag, and at least one fourth restriction flag.

[185] FIG. 14A-14B show an exemplary structure (1400) of general constraint information syntax according to an embodiment of the disclosure. A common constraint information syntax structure (1400) may represent constraint information (eg, referred to as common constraint information). The general constraint information (or general constraint information syntax structure (1400)) may include one or more constraint flags, such as a constraint flag (eg, general_frame_structure_constraint_group_flag) (1401), a constraint flag (eg, high_level_functionality_constraint_group_flag) (1402), a constraint flag (e.g. scalability_constraint_group_flag) (1403), control flag (e.g. partitioning_constraint_group_flag) (1404), control flag (e.g. intra_coding_tool_constraint_group_flag) (1405), control flag (e.g. inter_coding_tool_tool) (1406), control flag (e.g. transfom_contraint_group_flag) ( 1407), control flag (eg, inloop_filtering_constraint_group_flag) (1408) in Fig. 14A. One or more control flags (eg, flags (1401) - (1408)) may be present at the beginning of the general restriction information syntax structure (1400), as shown in FIG. 14A.

[186] The regulation flag (eg, general_frame_structure_constraint_group_flag) (1401) is associated with the constraint information group (1410) and is associated with the constraint flags (1411) - (1414), which are in the constraint information group (1410). A constraint flag (eg, general_frame_structure_constraint_group_flag) (1401) of 1 may indicate that constraint flags (1411) to (1414) may be present, which are in the constraint information group (1410).

[187] The constraint information group (1410) (or constraint flags (1411) - (1414) may be associated with the input source and frame packaging (eg, packed or projected frame). Referring to FIG. 14A, the constraint flags (1411) to (1414) correspond to the general_non_packed_constraint_flag (1411), general_frame_only_constraint_flag (1412), general_non_projected_constraint_flag (1413), and general_one_picture_only_constraint_flag (1414). Otherwise, a constraint flag (eg, general_frame_structure_constraint_group_flag) (1401) of 0 may indicate that the constraint flags (1411) - (1414) that are in the constraint information group (1410) may not be present.

[188] Referring to FIG. 14B, a control flag (eg, high_level_functionality_constraint_group_flag) (1402) of 1 may indicate that there may be constraint flags related to high-level functionality (eg, reference picture resampling) that are in the group. (1420) restriction information. Otherwise, a constraint flag (eg, high_level_functionality_constraint_group_flag) (1402) of 0 may indicate that the constraint flags that are in the constraint information group (1420) may be missing.

[189] Referring back to FIG. 14A, a control flag (eg, scaleability_constraint_group_flag) (1403) of 1 may indicate that constraint flag(s) related to scalability (eg, inter-layer prediction) may be present. Otherwise, the scalability-related constraint flag(s) may be missing.

[190] A constraint flag (eg, partitioning_constraint_group_flag) (1404) of 1 may indicate that constraint flag(s) related to high-level partitioning (eg, sub-image or tile) may be present. Otherwise, restriction flags related to high-level partitioning may be missing.

[191] A control flag (eg, intra_coding_tool_constraint_group_flag) (1405) of 1 may indicate that constraint flag(s) related to intra-coding (eg, intra-prediction) may be present. Otherwise, the restriction flag(s) related to internal encoding may be missing.

[192] A constraint flag (eg, inter_coding_tool_constraint_group_flag) (1406) of 1 may indicate that constraint flag(s) related to inter-coding (eg, motion compensation for inter-frame prediction) may be present. Otherwise, restriction flags related to intercoding may be missing.

[193] A control flag (eg, transfom_contraint_group_flag) (1407) of 1 may indicate that constraint flag(s) related to transform encoding (eg, multiple transform matrices) may be present. Otherwise, restriction flags related to transform encoding may be missing.

[194] In the embodiment, when all control flags (eg, control flags (1401) - (1408) in FIG. 14A) are 0, there are no flags in the constraint information syntax structure (eg, common constraint information syntax structure (1400)). restrictions.

[195] According to aspects of the disclosure, the syntax may be designed such that the control information includes control flags (e.g., control flags (1401) - (1408)), associated limit flags (e.g., limit flags (1411) - (1412) and restriction flags in the restriction information group (1420), additional control information and/or the like. may be byte aligned, for example the number of flags is a multiple of 8 to maintain byte alignment. In one example, the number of control flags and restriction flags in the restriction information (eg, in the syntax structure (1400) of the general restriction information) is a multiple of 8. A byte alignment mechanism can be used to achieve byte alignment of the control information. Referring to FIG. 14B, a syntax (eg, while loop) (1430) may be used for byte alignment.

[196] In an embodiment, offset information such as an offset (e.g., a constraint_info_offset[]) syntax element), and length information such as a length (e.g., a constraint_info_length[] syntax element) are present in the constraint information (e.g., at the beginning general restriction information syntax structure) indicating that at least one restriction flag in the restriction information group of the control flag is present in the restriction information. In an embodiment, one or more of the at least one group of constraint information is present in the encoded video bitstream. The offset and length may be present in the constraint information, indicating that at least one constraint flag in each of one or more of the at least one constraint information group is present in the constraint information. The number of offsets may be signaled in the encoded bit stream.

[197] In an embodiment, zero or more offsets (or constraint information offsets) (eg, denoted by the constraint_info_offset[] syntax element) and corresponding zero or more lengths (or constraint information lengths) (eg, denoted by the constraint_info_length[] syntax element) may be present in the constraint information, for example, at the beginning of the general constraint information syntax structure. The number of zero or more constraint information offsets and the number of zero or more constraint information lengths may be explicitly specified by syntax element(s).

[198] In the example, the number of zero or more constraint information offsets and the number of zero or more constraint information lengths are identical, and the number of zero or more constraint information offsets (represented by M) is explicitly indicated (or signaled) by a syntax element (eg, num_constaint_info_set). M may be an integer greater than or equal to 0. A value of M equal to 0 may indicate that there are no offsets of the constraint information and length of the constraint information in the constraint information.

[199] In an embodiment, the constraint information offset (e.g., constraint_info_offset[i] syntax element) and the constraint information length (e.g., constraint_info_length[i] syntax element) may indicate available constraint flags that are present in the constraint information (e.g., a common syntax structure restriction information). In an example where the constraint information offset value (eg, constraint_info_offset[i] syntax element) is 5 and the constraint information length value (eg, constraint_info_length[i] syntax element) is 3, the fifth, sixth, and seventh constraint flags are present in the constraint information (for example, in the structure of the general information syntax restrictions).

[200] In the example, if M is 0, the constraint flags are not present in the constraint information (eg, in the general constraint information syntax structure).

[201] In an example, run length encoding may be used to encode restriction flags that are specified in a predetermined order (or specified order).

[202] In an embodiment, run length encoding may be used where the constraint flags are specified in a predetermined order (or specified order). Instead of encoding the constraint flags directly, an appropriately encoded list of "skip" values can specify constraint flags that are zero, with the next constraint flag implied to be 1. The run length encoding described above can be particularly effective if (i ) the number of constraint flags is large and (ii) a small percentage of constraint flags are 1.

[203] In an embodiment, one or more of at least one group of constraint information is present in the encoded video bitstream. A plurality of restriction flags in one or more of the at least one group of restriction information are signaled in accordance with a predetermined order. Accordingly, a plurality of constraint flags may be encoded using run-length encoding (eg, run-length encoding or run-length decoding). In addition, prediction information for a subset of coding blocks may be determined based on a plurality of restriction flags.

[204] In an embodiment, the at least one restriction flag in the control flag restriction information group includes a plurality of restriction flags signaled in accordance with a predetermined order. Accordingly, a plurality of constraint flags may be encoded using run-length encoding (eg, run-length encoding or run-length decoding).

[205] In an embodiment, a complete list of restriction flags may be defined in a video coding standard (eg, the VVC specification), an external table, or the like. In the example, only the available constraint flags of the constraint flags are indicated, for example, by one or more of the following: the number of available constraint flags (eg, num_available_constraint_flags), the constraint flag(s) (or the presence flag(s) of a constraint group), the constraint information offset information, and the constraint flag(s) information. length of information restrictions, etc. present in the encoded video stream.

[206] In the example, a complete list of restriction flags is specified and made available to the encoder and decoder. A complete list of restriction flags can be stored in the decoder. A complete list of limit flags may include 100 limit flags. 10 of the 100 restriction flags are present in the restriction information for CLVS and are thus available to a subset of coding blocks in CLVS. 10 out of 100 limit flags are called 10 available limit flags. The example signals the number of available limit flags (eg 10). In the example, the 10 available restriction flags are in two groups of restriction information and are governed by a first throttling flag and a second throttling flag. Thus, the first throttling flag and the second throttling flag can be signaled to indicate the 10 limit flags available.

[207] The example signals a first constraint information offset (eg, constraint_info_offset[0] syntax element) and a first constraint information length (eg, constraint_info_length[0] syntax element). A second constraint information offset (eg, constraint_info_offset[1] syntax element) and a second constraint information length (eg, constraint_info_length[1] syntax element) are signaled. For example, the constraint_info_offset[0] syntax element is 15, and the constraint_info_length[0] syntax element is 3, the constraint_info_offset[1] syntax element is 82, and the constraint_info_length[1] syntax element is 7, and thus indicates that with 15 The th to 17th restriction flags and the 82nd to 88th restriction flags in the complete list (eg, 100 restriction flags) are available or present in the restriction information.

[208] In an embodiment, any of various methods (or methods, embodiments, examples) for efficiently encoding constraint flags can be combined using suitable control information. The combination may be a suitable combination of two or more such methods. Alternatively, one of the various methods (or methods, embodiments, examples) can be used independently. Restriction flags can be grouped. A certain group(s) may use run length encoding while another group(s) may use straight binary encoding.

[209] The value of the maximum number of constraint flags (eg, MaxNumConstraintFlags) may be predetermined to be 16, 32, 64, 128, or the like.

[210] The value of the maximum number of constraint flags (eg, MaxNumConstraintFlags) may be determined by profile information such as general_profile_idc or general_sub_profile_idc, or codec version information, such that the range of the number of constraint flags (eg, num_available_constraint_flags (1301)) may be limited by the profile information or version information. For example, the value for the number of constraint flags (for example, num_available_constraint_flags (1301)) in the main profile (for example, where MaxNumConstraintFlags=64) can range from 0 to 64, while the value for the number of constraint flags (for example, num_available_constraint_flags (1301)) in an extended profile (for example, where MaxNumConstraintFlags=128) can range from 0 to 128.

[211] In an embodiment, the value of the number of constraint flags (eg, num-available_constraint_flags) may be inferred to be equal to a value predetermined by profile information such as general_profile_idc or general_sub_profile_idc, or codec version information, so that the value of num_available_constraint_flags can be determined without explicit signaling.

[212] In the embodiment shown in FIG. 12, information about the reserved bytes may be present in the general constraint information syntax structure. For example, the gci_num_reserved_bytes (1203) and gci_reserved_bytes[] (1204) flags may be present in the common constraint information syntax structure to extend the common constraint information syntax structure. The gci_num_reserved_bytes flag can specify the number of limit bytes reserved. In one example, the reserved constraint bytes are intended to signal additional flags (eg, additional constraint flags). The gci_reserved_byte[] flag can have any appropriate value.

[213] In an embodiment, the value of gci_num_reserved_bytes may be limited or determined by profile information such as general_profile_idc or general_sub_profile_idc, or codec version information. With a basic profile (or primary profile), the value of the gci_num_reserved_bytes flag can be 0. With an advanced profile (or advanced profile), the value of gci_num_reserved_bytes can be greater than 0.

[214] A field sequence flag may be signaled in the encoded video bitstream. The field sequence flag may indicate whether the images in the output layer are encoded using field encoding. The field sequence flag may be signaled in SPS as sps_field_seq_flag. In an embodiment, the sps_field_seq_flag may be present in the SPS. The sps_field_seq_flag equal to 1 may indicate that CLVS transmits images that represent fields. An sps_field_seq_flag of 0 may indicate that CLVS transmits images that represent frames.

[215] In the general constraint information syntax structure of FIG. 12, a general_frame_only_constraint_flag may be present. A general_frame_only_constraint_flag equal to 1 can indicate that the scope for a set of output layers (for example, OlsInScope) conveys images that represent frames. A general_frame_only_constraint_flag of 0 indicates that the scope for a set of output layers (such as OlsInScope) conveys images that may or may not represent frames. In an embodiment, the general_frame_only_constraint_flag indicates whether the images in the set of output layers are field encoded. The set of output layers may include a subset of encoding blocks. The sps_field_seq_flag may be false based on the general_frame_only_constraint_flag (e.g., set to 1) indicating that the subset of images is not field encoded; the subset of images may be on the same layer in the set of output layers.

[216] When general_frame_only_constraint_flag is 1, the value of sps_field_seq_flag can be 0.

[217] In an embodiment, the pps_mixed_nalu_types_in_pic_flag flag may be present in the PPS. A pps_mixed_nalu_types_in_pic_flag of 1 may indicate that each picture referencing a PPS has more than one VCL NAL unit, and the VCL NAL units do not have the same nal_unit_type value. A pps_mixed_nalu_types_in_pic_flag equal to 0 may indicate that each picture referencing a PPS has one or more NAL VCL units, and the NAL VCL units of each image referencing a PPS have the same nal_unit_type value. In the general structure of the constraint information syntax of FIG. 12, a no_mixed_nalu_types_in_pic_constraint_flag may be present. A no_mixed_nalu_types_in_pic_constraint_flag of 1 may indicate that the value of pps_mixed_nalu_types_in_pic_flag should be 0. A no_mixed_nalu_types_in_pic_constraint_flag of 0 does not impose such a constraint.

[218] In an embodiment, the general_one_picture_only_constraint_flag may be present in the general constraint information syntax structure of FIG. 12. A general_one_picture_only_constraint_flag value of 1 may indicate that there is only one coded picture in the bitstream. The general_one_picture_only_constraint_flag equal to 0 does not impose such a restriction.

[219] In an embodiment, the single_layer_constraint_flag may be present in the general constraint information syntax structure of FIG. 12. A single-layer_constraint_flag equal to 1 may indicate that the sps_video_parameter_set_id should be equal to 0. A single_layer_constraint_flag equal to 0 does not impose such a constraint. When the general_one_picture_only_constraint_flag is 1, the single_layer_constraint_flag can be set to 1.

[220] In an embodiment, the all_layers_independent_constraint_flag may be present in the general constraint information syntax structure of FIG. 12. An all_layers_independent_constraint_flag equal to 1 may indicate that the vps_all_independent_layers_flag may be equal to 1. An all_layers_independent_constraint_flag equal to 0 does not impose such a constraint. When single_layer_constraint_flag is 1, the value of all_layers_independent_constraint_flag can be 1.

[221] In an embodiment, the no_res_change_in_clvs_constraint_flag flag may be present in the general constraint information syntax structure of FIG. 12. A no_res_change_in_clvs_constraint_flag equal to 1 may indicate that the sps_res_change_in_cfvs_allowed_flag may be equal to 0. A no_res_change_in_clvs_constraint_flag equal to 0 does not impose such a restriction. When no_ref_pic_resampling_constraint_flag is 1, the value of no_res_change_in_clvs_constraint_flag can be 1.

[222] In an embodiment, the no_mixed_nalu_types_in_pic_constraint_flag flag may be present in the general constraint information syntax structure of FIG. 12. A no_mixed_nalu_types_in_pic_constraint_flag of 1 indicates that the pps_mixed_nalu_types_in_pic_flag can be set to 0. A no_mixed_nalu_types_in_pic_constraint_flag of 0 does not impose such a constraint. When the one_subpic_per_pic_constraint_flag is 1, the value of the no_mixed_nalu_types_in_pic_constraint_flag can be 1.

[223] In an embodiment, the no_trail_constraint_flag may be present in the general constraint information syntax structure of FIG. 12. The no_trail_constraint_flag equal to 1 may indicate that OlsInScope may not have a NAL unit with nuh_unit_type equal to TRAIL_NUT. The no_trail_constraint_flag, set to 0, does not impose such a constraint. When the general_one_picture_only-constraint_flag is 1, the no_trail_constraint_flag can be 1.

[224] In an embodiment, the no_stsa_constraint_flag may be present in the overall constraint information syntax structure of FIG. 12. The no_stsa_constraint_flag equal to 1 may indicate that OlsInScope may not have a NAL unit with nuh_unit_type equal to STSA_NUT. The no_stsa_constraint_flag of 0 does not impose such a constraint. When the general_one_picture_only_constraint_flag is 1, the no_stsa_constraint_flag can be 1.

[225] In an embodiment, the no_trail_constraint_flag may be present in the general constraint information syntax structure of FIG. 12. The no_trail_constraint_flag equal to 1 may indicate that OlsInScope may not have a NAL unit with nuh_unit_type equal to TRAIL_NUT. The no_trail_constraint_flag, set to 0, does not impose such a constraint. When the general_one_picture_only_constraint_flag is 1, the no_trail_constraint_flag can be 1.

[226] In an embodiment, the no_stsa_constraint_flag may be present in the overall constraint information syntax structure of FIG. 12. The no_stsa_constraint_flag equal to 1 may indicate that OlsInScope may not have a NAL unit with nuh_unit_type equal to STSA_NUT. The no_stsa_constraint_flag of 0 does not impose such a constraint. When the general_one_picture_only_constraint-flag is 1, the no_stsa_constraint_flag can be 1.

[227] In an embodiment, the no_idr_constraint_flag may be present in the general constraint information syntax structure of FIG. 12. A no_idr_constraint_flag equal to 1 may indicate that OlsInScope may not have a NAL unit with nuh_unit_type equal to IDR_W_RADL or IDR_N_LP. The no_idr_constraint_flag of 0 does not impose such a constraint.

[228] In an embodiment, the no era constraint flag may be present in the general constraint information syntax structure of FIG. 12. A flag_no_cra_constraint_flag equal to 1 may indicate that OlsInScope may not have a NAL unit with nuh_unit_type equal to CRANUT. The no_cra_constraint_flag of 0 does not impose such a constraint.

[229] In an embodiment, the no_rasl_constraint_flag may be present in the general constraint information syntax structure of FIG. 12. A no_rasl_constraint_flag equal to 1 may indicate that OlsInScope may not have a NAL unit with nuh_unit_type equal to RASL_NUT. The no_rasl_constraint_flag of 0 does not impose such a constraint. When no_cra_constraint_flag is 1, the value of no_rasl_constraint_flag can be 1.

[230] In an embodiment, the no_radl_constraint_flag may be present in the general constraint information syntax structure of FIG. 12. A no_radl_constraint_flag equal to 1 may indicate that OlsInScope may not have a NAL unit with nuh_unit_type equal to RADL_NUT. The no_radl_constraint_flag of 0 does not impose such a constraint. When no_idr_constraint_flag is 1 and no_cra_constraint_flag is 1, the value of no_rasl_constraint_flag can be 1.

[231] FIG. 15A-15D show an exemplary structure (1500) of general constraint information syntax according to an embodiment of the disclosure. A common constraint information syntax structure (1500) may represent constraint information (eg, referred to as common constraint information). The common constraint information (or common constraint information syntax structure (1500)) may include a flag (eg, gci_present_flag) indicating whether constraint flags are present in the common constraint information syntax structure (1500). In one example, if a flag (eg, gci_present_flag) is true, one or more constraint flags may be present in the constraint general information syntax structure (1500). If the flag (eg, gci_present_flag) is false, no constraint flags can be present in the constraint general information syntax structure (1500).

[232] As described above with reference to FIGS. 14A-14B, the restriction flags of FIGS. 15A-15D can be classified into groups (eg, restriction information groups). Whether restriction flags in a restriction information group can be present in the general restriction information syntax structure (1500) can be controlled, for example, by using a corresponding control flag for the restriction information group.

[233] The general constraint information syntax structure (1500) may include any suitable number of constraint information groups. In an example, the general constraint information syntax structure (1500) may further include constraint flag(s) that are not included in the constraint information groups, and thus the constraint flag(s) are outside the constraint information groups.

[234] In the example shown in FIGS. 15A-15D, the general constraint information syntax structure (1500) includes nine constraint information groups (1510) to (1518). The constraint information group (1510) is associated with general constraint information and includes three constraint flags (eg, gci_intra_only_constraint_flag, gci_all_layers_indepedent_constraint_flag, gci_one_au_only_constraint_flag). In one example, a control flag may regulate or control whether a group of constraint information (1510) is present in a general constraint information syntax structure (1500).

[235] The restriction information group (1511) and corresponding restriction flags relate to image format restriction information. In one example, a control flag may regulate or control whether a group (1511) of constraint information is present in the overall structure of the constraint information syntax (1500).

[236] The restriction information group (1512) and corresponding restriction flags are associated with the NAL unit type restriction information. In one example, a control flag may regulate or control whether a group of restriction information (1512) is present in the general restriction information syntax structure (1500).

[237] A restriction information group (1513) and corresponding restriction flags relate to tile, slice, or sub-image division restriction information. In one example, a control flag may regulate or control whether a group of constraint information (1503) is present in a general constraint information syntax structure (1500).

[238] A constraint information group (1514) and corresponding constraint flags are associated with the CTU and block division constraint information. In one example, a control flag may regulate or control whether a group of restriction information (1514) is present in the general restriction information syntax structure (1500).

[239] The restriction information group (1515) and corresponding restriction flags refer to intra-coding restriction information. In one example, a control flag may regulate or control whether a group of constraint information (1515) is present in the general constraint information syntax structure (1500).

[240] The restriction information group (1516) and corresponding restriction flags refer to inter-coding restriction information. In one example, a control flag may regulate or control whether a group of restriction information (1516) is present in the general restriction information syntax structure (1500).

[241] The constraint information group (1517) and corresponding constraint flags are associated with the transform, quantization, and remainder constraint information. In one example, a control flag may regulate or control whether a group of constraint information (1517) is present in a general constraint information syntax structure (1500).

[242] The restriction information group (1518) and the corresponding restriction flags refer to the loop filter restriction information. In one example, a control flag may regulate or control whether a group of constraint information (1518) is present in the general constraint information syntax structure (1500).

[243] In the embodiment shown in FIG. 15D, syntax elements gci_num_reserved_bits and gci_reserved_zero_bit[] may be present in the common constraint information syntax structure (1500) to extend the common constraint information syntax structure. The gci_num_reserved_bits syntax element can specify the number of constraint bits to be reserved. The gci_reserved_zero_bit[] syntax element can have any suitable value. In the example, the presence of the gci_reserved_zero_bit[] syntax element and the value of gci_reserved_zero_bit[] do not affect the decoding process specified in a particular video encoding standard (for example, the VVC specification). The presence of the gci_reserved_zero_bit[] syntax element and the gci_reserved_zero_bit[] value can be used for some video encoding standards.

[244] FIG. 16 shows an example of a set of PTL syntax elements in a PTL syntax structure according to an embodiment of the disclosure. The set of PTL syntax elements may include general_profile_idc, general_tier_flag, general_level_idc, ptl_frame_only_constraint_flag, ptl_multilayer_enabled_flag, plt_sublayer_level_present_flag[], ptl_reserved_zero_bit, sublayer_level_idc[], ptl_num_sub_profiles, general_sub_profile_idc, and/or general restriction information. Whether some (eg general_profile_idc, general_tier_flag, general constraint information, ptl_num_sub_profiles) set of PTL syntax elements can be present can be based on flag values (eg profileTierPresentFlag).

[245] A video sequence with one or more layers may be encoded as a field encoding structure, where two or more consecutive field images may contain a full frame image. Field encoding information may be signaled in a high-level syntactic structure such as parameter sets or SEI messages.

[246] In an embodiment, each frame of interlaced video has two fields for each frame. In some examples, you can use a three-two conversion (3:2 downscaling) to convert 24 fps to 29.97 or 30 fps. A 3:2 reduction can convert four frames into five frames, plus a slight speed reduction, roughly.

[247] In an embodiment, a flag (eg, sps_field_seq_flag) may be signaled in the SPS, as shown in FIG. 17 to indicate whether field encoding is used or not. A flag (eg sps_field_seq_flag) equal to 1 may indicate that CLVS transmits images that represent fields. A flag (eg sps_field_seq_flag) equal to 0 may indicate that CLVS transmits images that represent frames. When a flag (for example, general_frame_only_constraint_flag) is 1, the value of a flag (for example, sps_field_seq_flag) can be 0. In the example, a flag (for example, general_frame_only_constraint_flag) that is 1 indicates that the value of a flag (for example, sps_field_seq_flag) is 0.

[248] In an embodiment, when the flag (eg, sps_field_seq_flag) is 1, an SEI message with frame field information may be present for each coded picture in the CLVS. This presence may be required by a video encoding standard or technology.

[249] In an embodiment, the decoding process may not treat images that represent fields or frames differently than other images. A sequence of images representing fields may be encoded with the dimensions of the individual field image. For example, images that represent 1080i fields may have cropped output dimensions of 1920x540, while the sequence image rate may express the frequency of the source fields (for example, typically 50 to 60 Hz) instead of the source frame rate (for example, typically between 25 and 30 Hz).

[250] In an embodiment, the value of the flag (eg, sps_field_seq_flag) may be the same for all layers. In one example, the value of a flag (eg sps_field_seq_flag) may be the same in all SPS referenced by CLVS in CVS, for example, because all layers may have the same field structure or the same frame structure.

[251] In an embodiment, the value of the flag (eg, sps_field_seq_flag) of layer A is equal to the value of the flag (eg, sps_field_seq_flag) of the dependent layer in layer A in the set of output layers.

[252] In an embodiment, when a flag (eg, sps_field_seq_flag) is 1, all fields that include the same frame may belong to the same PPS, so that all fields of the same frame may have the same image size , matching window and tiling.

[253] In an embodiment, two frame fields may have the same partitioning structure with the same tile syntax elements or the same sub-image syntax elements.

[254] In an embodiment, the two fields of a frame may have an identical layer structure with the same number of layers and sublayers.

[255] In the embodiment shown in FIG. 18, a flag (eg, vps_field_seq_flag) may be signaled to the VPS to indicate whether layers referencing the VPS are encoded as a field or a frame. A flag (eg, vps_field_seq_flag) equal to 1 may indicate that the CVS referencing the VPS transmits images that represent fields. A flag (eg, vps_field_seq_flag) equal to 0 may indicate that the CVS referencing the VPS transmits images that represent frames. When a flag (for example, general_frame_only_constraint_flag) is 1, the value of the flag (for example, sps_field_seq_flag) must be 0.

[256] When a flag (eg, vps_field_seq_flag) is present in a VPS, the value of the flag (eg, sps_field_seq_flag) in the SPS referencing the VPS may be equal to the value of the flag (eg, vps_field_seq_flag).

[257] High-level syntax cleanup in SPS

[258] The following high-level syntax sanitization (HLS) may be performed in a sequence parameter set (SPS) according to embodiments of the present disclosure: (1) sps_picture_header_in_slice_header_flag signaling; (2) Align the subpic_treated_as_pic_flag value across layers; (3) Limit on the number of sub-images for a small image; (4) Correction of sps_subpic_id_len_minus1 value range.

[259] 1. Picture_header_in_slice_header flag signaling

[260] When the image header is contained in the slice header, it is assumed that the sub-image information is not used and only one slice is present per image in the CLVS referencing the Sequence Parameter Set (SPS). This assumption has been hypothetically agreed upon, but not clearly described. To clean up HTTP live streaming, you can define and use the new sps_picture_header_in_slice_header_flag in SPS. In this case, sps_picture_header_in_slice_header_flag equal to 1 indicates that the picture header (PH) syntax structure is present in the slice header, and only one slice is present per image in CLVS. Additionally, there is no sub-picture information if sps_picture_header_in_slice_header_flag is 1.

[261] The following are examples of semantic and syntactic definitions for the new sps_picture_header_in_slice_header_flag:

[272] sps_picture_header_in_slice_header_flag equal to 1 indicates that the PH syntax structure is present in the slice header, and only one slice is present per image in CLVS. However, sps_picture_header_in_slice_header_flag equal to 0 indicates that the PH syntax structure is not present in the slice header in CLVS.

[273] subpic_info_present_flag equal to 1 indicates that subpicture information is present for the CLVS, and there can be one or more subpictures in each CLVS picture. A subpic_info_present_flag of 0 indicates that there is no subpicture information for the CLVS and that there is only one subpicture in each CLVS picture. If missing, subpic_info_present_flag is output as 0.

[274] When res_change_in_clvs_allowed_flag is 1, the value of subpic_info_present_flag must be 0.

[275] In one embodiment, picture_header_in_slice_header_flag is signaled in the image header.

[276] picture_header_in_slice_header_flag equal to 1 indicates that the PH syntax structure is present in the slice header. In this case, picture_header_in_slice_header_flag equal to 0 indicates that the PH syntax structure is not present in the slice header.

[277] The value of picture_header_in_slice_header_flag must be equal to sps_picture_header_in_slice_header_flag.

[278] When picture_header_in_slice_header_flag is 1 for a coded slice, the bitstream compliance requirement is that a VCL NAL unit with nal_unit_type equal to the NUT PH must not be present in the CLVS.

[279] When picture_header_in_slice_header_flag is 0, all encoded slices in the current picture shall have picture_header_in_slice_header_flag equal to 0, and the current PU shall have a NAL PH unit.

[280] Additionally, the constant flag pic_header_in_slice_header_constraint_flag can be added to general constraint_info(). When pic_header_in_slice_header_constraint_flag is 1, the value of picture_header_in_slice_header_flag in the PH must be 1.

[281] pic_header_in_slice_header_constraint_flag equal to 1 indicates that the PH syntax structure is present in the slice header. pic_header_in_slice_header_constraint_flag equal to 0 does not impose such a constraint. When pic_header_in_slice_header_constraint_flag should be 1, picture_header_in_slice_header_flag should be 1.

[282] When pic_header_in_slice_header_constraint_flag is 1, rpl_info_in_ph_flag, dbf_info_in_ph_flag, sao_info_in_ph_flag, wp_info_in_ph_flag, qp_delta_info_in_ph_flag must be 0.

[283] When pic_header_in_slice_header_constraint_flag is 1, rect_slice_flag must be 1.

[284] When pic_header_in_slice_header_constraint_flag is 1, subpic_info_present_flag must be 0.

[285] When pic_header_in_slice_header_constraint_flag is 1, separate_colour_plane_flag must be 0.

[286] The following is an example of defining the syntax for the new restriction flag:

[287] The following is an example of defining the semantics for the new restriction flag:

[288] separate_colour_plane_flag equal to 1 indicates that the three color components of the 4:4:4 chroma format are encoded separately. A separate_colour_plane_flag of 0 indicates that color components are not encoded separately. If separate_colour_plane_flag is absent, it is assumed to be 0. When separate_colour_plane_flag is 1, the encoded image consists of three separate components, each consisting of encoded samples of a single color plane (Y, Cb, or Cr) and using monochrome encoding syntax. In this case, each color plane is associated with a specific color_plane_id value.

[289] NOTE: - In decoding processes there is no dependency between color planes having different color_plane_id values. For example, the process of decoding a monochrome image with one color_plane_id value does not use any data from monochrome images having different color_plane_id values for inter-frame prediction.

[290] Depending on the value of separate_colour_plane_flag, the value of the ChromaArrayType variable is assigned as follows:

[291] - If separate_colour_plane_flag is 0, ChromaArrayType is set to chroma_format_idc.

[292] - Otherwise (separate_colour_plane_flag is 1) ChromaArrayType is set to 0.

[293] When pic_header_in_slice_header_constraint_flag is 1, separate_colour_plane_flag must be 0.

[294] subpic_info_present_flag equal to 1 indicates that subpicture information is present for the CLVS, and there can be one or more subpictures in each CLVS picture. A subpic_info_present_flag of 0 indicates that there is no subpicture information for the CLVS and that there is only one subpicture in each CLVS picture.

[295] When res_change_in_clvs_allowed_flag is 1, the value of subpic_info_present_flag must be 0.

[296] NOTE: - When a bitstream is the result of a sub-bitstream extraction process and contains only a subset of subpictures of the incoming bitstream for the sub-bitstream extraction process, it may be necessary to set the value of subpic_info_present_flag to 1 in the RBSP SPS.

[297] When pic_header_in_slice_header_constraint_flag is 1, subpic_info_present_flag must be 0.

[298] rect_slice_flag equal to 0 indicates that the tiles in each slice are in raster scan order, and slice information is not signaled to the PPS. A rect_slice_flag of 1 indicates that the tiles within each slice cover a rectangular area of the image, and the slice information is signaled in the PPS. If absent, rect_slice_flag is assumed to be 1. When subpic_info_present_flag is 1, the value of rect_slice_flag must be 1.

[299] When pic_header_in_slice_header_constraint_flag is 1, rect_slice_flag must be 1.

[300] qp_delta_info_in_ph_flag equal to 1 indicates that QP delta information is present in the PH syntax structure and is not present in slice headers referencing PPSs that do not contain the PH syntax structure. A qp_delta_info_in_ph_flag equal to 0 indicates that QP delta information is not present in the PH syntax structure and may be present in slice headers referencing PPSs that do not contain the PH syntax structure.

[301] When pic_header_in_slice_header_constraint_flag is 1, rpl_info_in_ph_flag, dbf_info_in_ph_flag, sao_info_in_ph_flag, wp_info_in_ph_flag, qp_delta_info_in_ph_flag must be 0.

[302] one_subpic_per_pic_constraint_flag equal to 1 specifies that each picture should contain only one subpicture, one_subpic_per_pic_constraint_flag equal to 0 does not impose such a constraint. When one_slice_per_pic_constraint_flag is 1, the value of one_subpic_per_pic_constraint_flag must be 1.

[303] pic_header_in_slice_header_constraint_flag equal to 1 indicates that the PH syntax structure is present in the slice header. pic_header_in_slice_header_constraint_flag equal to 0 does not impose such a constraint. When pic_header_in_slice_header_constraint_flag should be 1, picture_header_in_slice_header_flag should be 1.

[304] 2. Aligning the subpic_treated_as_pic_flag value across layers

[305] To extract a sub-image from a multi-layer bitstream when subpic_treated_as_pic_flag[i] is equal to 1, for each output layer and its reference layers in OLS, all images in the output layer and its reference layers must have a value of subpic_treated_as_pic_flag[j] equal to up to 1 for each value of j in the range from 0 to sps_num_subpics_minus1. Therefore, according to embodiments of the present disclosure, the following limitation is added.

[306] subpic_treated_as_pic_flag [i] equal to 1 indicates that the i-e subpicture of each coded picture in CLVS is treated as an image during the decoding process, excluding intra-loop filtering operations.

subpic_treated_as_pic_flag [i] equal to 0 indicates that the i-oe subpicture of each encoded picture in CLVS is not treated as an image during the decoding process, excluding intra-loop filtering operations. If missing, the value of subpic_treated_as_pic_flag[i] is assumed to be equal to sps_independent_subpics_flag.

[307] When subpic_treated_as_pic_flag[i] is 1, bitstream matching requires all of the following conditions to be true for each output layer and its reference layers in the OLS, which includes the layer containing the i-oe subimage as the output layer:

[308] - All images in the output layer and its reference layers must have the same pic_width_in_luma_samples value and the same pic_height_in_luma_samples value.

[309] - All SPS referenced by the output layer and its reference layers must have the same value of sps_num_subpics_minus1 and must have the same values of subpic_ctu_top_left_x [j], subpic_ctu_top_left_y [j], subpic_width_minus1 [j], subpic_fil_subpics_minus1 and subpic_height_subpics_minus1, [ j], respectively, for each value of j in the range from 0 to sps_num_subpics_minus1 inclusive.

[310] - All images in each access unit in the output layer and its reference layers must have the same SubpicIdVal[j] value for each value of j in the range 0 to sps num subpics minus1 inclusive.

[311] - All images in the output layer and its reference layers must have a subpic_treated_as_pic_flag[j] value of 1 for each value of j between 0 and sps_num_subpics_minus1 inclusive.

[312] 3. Limitation on the number of sub-images for a small image

[313] In one embodiment, an additional constraint may be imposed such that the number of image tiles must be 1 when the image size does not exceed CtbSizeY.

[314] The following are examples of syntax and/or semantic definitions for further limitation:

[315] When pic_width_max_in_luma_samples is not greater than CtbSizeY and pic_height_max_in_luma_samples is not greater than CtbSizeY, the value of sps_num_subpics_minus1 shall be 0.

[316] 4. Adjusting the range of values sps_subpic_id_len_minus1

[317] In one embodiment, the semantics of sps_subpic_id_len_minus1 can be modified as follows so that the syntax element can have a more precise range of values.

[318] sps_subpic_id_len_minus1 plus 1 specifies the number of bits used to represent the sps_subpic_id[i] syntax element, the pps_subpic_id[i] syntax elements, if any, and the slice_subpic_id syntax element, if any. The sps_subpic_id_len_minus1 value must be between 0 and 15, inclusive. The value 1<<(sps_subpic_id_len_minus1) must be greater than or equal to sps_num_subpics_minus1 +1.

[319] The above embodiments can be used individually or combined in any order. In addition, each of the encoder and/or decoder embodiments may be implemented by processing circuits (eg, one or more processors or one or more integrated circuits). In one example, one or more processors executes a program that is stored on a non-transitory computer-readable medium.

[320] In FIG. 19 is a flow diagram describing a process (1900) according to an embodiment of the invention. Process (1900) may be used in block reconstruction to generate a prediction block for the block being reconstructed. The term "block" in the disclosure may be interpreted as a prediction block, CB, CU, or the like. In various embodiments, the process (1900) is performed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330) and (340), wherein the processing circuitry functions as a video encoder (403), and the processing circuitry functions as a video decoder ( 410), the processing circuit functions as a video decoder (510), the processing circuit functions as a video encoder (603), etc. In some embodiments, the process (1900) is implemented in program instructions, such that when the processing circuit executes the program instructions, the processing circuit executes the process (1900). The process starts from (S1901) and proceeds to (S1910).

[321] At (S1910), there may be a control flag obtained for each of the at least one restriction information group in the restriction information signaled in the encoded video bitstream, each control flag may indicate whether a restriction information group of the at least one group is present restriction information corresponding to the corresponding control flag, in the restriction information, the restriction information may refer to a subset of encoding blocks in the encoded video bitstream.

[322] Constraint information may be present in high-level syntax (eg, SPS, VPS, DCI, DPS) for at least one picture, and a subset of encoding blocks may include encoding blocks in at least one picture.

[323] In the example, each control flag indicates that the group of restriction information corresponding to the corresponding control flag is not present in the restriction information, and there are no restriction flags in the restriction information.

[324] In (S1920), determining whether the control flag restriction information group is present in the restriction information, based on the control flag of the restriction information group of at least one restriction information group, the control flag restriction information group may include at least one restriction flag for a subset of coding blocks.

[325] In (S1930), determining prediction information for a subset of encoding blocks based on whether a control flag constraint information group is present in the constraint information.

[326] In the example, it is determined that a control flag restriction information group is present in the restriction information. The prediction information for a subset of coding blocks may be determined based on at least one restriction flag in the control flag restriction information group.

[327] In the example, it is determined that the control flag restriction information group is not present in the restriction information. The prediction information for a subset of coding blocks may be determined independently of the control flag restriction information group.

[328] At (S1940), reconstructing a subset of encoding blocks based on the prediction information.

[329] The process (1900) can be adapted accordingly. Step(s) in the process (1900) may be changed and/or omitted. Additional step(s) may be added. Any suitable implementation order may be used. For example, one or more of the at least one group of constraint information is present in the encoded video bitstream. A plurality of restriction flags in one or more of the at least one group of restriction information may be signaled in accordance with a predetermined order. A plurality of constraint flags can be decoded using run length decoding, and thus, prediction information for a subset of encoding blocks can be determined based on the plurality of constraint flags.

[330] In an example, the at least one restriction flag in the control flag restriction information group includes a plurality of restriction flags signaled in accordance with a predetermined order, and the plurality of restriction flags in the control flag restriction information group can be decoded using length decoding series.

[331] In an example, the offset and length are present in the constraint information, indicating that at least one constraint flag in the control flag constraint information group is present in the constraint information.

[332] For example, one or more of the at least one group of restriction information is present in the encoded video bitstream. The offset and length are present in the constraint information, indicating that at least one constraint flag in each of one or more of the at least one constraint information group is present in the constraint information. The number of offsets may be signaled in the encoded bit stream.

[333] For example, one or more of the at least one group of constraint information is present in the encoded video bitstream. One or more syntax elements in the constraint information may be byte aligned. The one or more syntax elements may include a control flag for each of the at least one group of restriction information and a plurality of restriction flags in one or more of the at least one group of restriction information.

[334] In an example, the at least one constraint flag includes a frame-only constraint flag (eg, general_frame_only_constraint_flag) indicating whether the image is encoded in the set of field-encoded output layers. The set of output layers may include a subset of encoding blocks. The field sequence flag (for example, sps_field_seq_flag) is false based on the frame-only limit flag indicating that the subset of images is not encoded using field encoding, where the subset of images is on the same layer of the output layer set.

[335] In one example, a field sequence flag is signaled in the encoded video bitstream, and a field sequence flag indicates whether the pictures in the output layer are encoded with field encoding.

[336] Embodiments of the present invention can be used individually or combined in any order. In addition, each of the encoder and decoder methods (or embodiments) may be implemented by processing circuitry (eg, one or more processors or one or more integrated circuits). In one example, one or more processors executes a program that is stored on a non-transitory computer-readable medium.

[337] Aspects of the disclosure relate to signaling field encoding information in a coded video stream, eg, with multiple layers.

[338] When images are encoded into a bitstream that includes multiple layers of different qualities, the bitstream may have syntax elements that indicate which layers can be output at the decoder side. In a particular video codec that supports multiple layers and scalability, one or more sets of output layers may be signaled to the VPS. Syntax elements specifying sets of output layers and their dependencies, profile/tier/layer, and reference model parameters of a hypothetical decoder must be effectively signaled in the parameter set. When multilayer video sequences are encoded as a field structure, where two or more field images include one frame image, the field encoding information may be signaled as a high-level syntactic structure such as an SPS, VPS, or SEI message.

[339] In some video encoding technologies, an access unit (AU) may refer to the encoded picture(s), slice(s), tile(s), NAL unit(s), and so on, that are captured and arranged into a corresponding picture/slice /tile/bitstream of the NAL unit at a given time. This point in time may be the time of composition.

[340] In some examples, such as HEVC, and some other video encoding technologies, a picture order counter (POC) value may be used to indicate a selected reference picture among a plurality of reference pictures stored in a decoded picture buffer (DPB). When an AU includes one or more images, slices or tiles, each image, slice or tile belonging to the same AU may have the same POC value, from which it can be inferred that one or more images, slices or tiles are created from content of the same layout time. Thus, two images/slices/tiles having the same set POC value may indicate that the two images/slices/tiles belong to the same AU and have the same layout time. Conversely, two images/slices/tiles having different POC values may indicate two images/slices/tiles that belong to different AUs and have different layout times.

[341] In an embodiment, the above tight relationship can be relaxed when the AU may contain images, slices or tiles with different POC values. By allowing different POC values within an AU, the POC values can be used to identify potentially independently decodable pictures/slices/tiles with identical presentation time. In this way, support for multiple scalable layers can be provided without changing the reference picture selection signaling (eg, reference picture set signaling or reference picture list signaling), as described in more detail below.

[342] In some examples, it is desirable to identify the AU that owns an image/slice/tile with respect to another image/slice/tile having different POC values based solely on the POC value. This can be achieved as described below.

[343] In embodiments, an access unit counter (AUC) may be signaled in a high-level syntactic structure such as a NAL unit header, a slice header, a tile group header, an SEI message, a parameter set, or an AU delimiter. The AUC value (or AUC value) can be used to identify which NAL units, images, slices or tiles belong to a given AU. The AUC value may correspond to a single point in time of the composition. The AUC value can be a multiple of the POC value. By dividing the POC value by an integer value, the AUC value can be calculated. In some examples, division operations may impose some overhead on decoder implementations. Thus, slight restrictions in the numbering space of AUC values may allow division operations to be replaced by shift operations. For example, the AUC value may be equal to the value of the most significant bit (MSB) of the POC value range.

[344] In an embodiment, the per-AU POC cycle value (eg, pos_cycle_au) may be signaled in a high-level syntactic structure such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or AU delimiter. The per-AU POC cycle value (eg, poc_cycle_au) may indicate how many different and consecutive POC values can be associated with the same AU. For example, if the POC cycle value per AU per AU (for example, POC_cycle_au) is 4, images, slices, or tiles with a POC value between 0 and 3, inclusive, are associated with an AU with an AUC value of 0, and images, slices, or tiles with POC values between 4 and 7 inclusive are associated with AUs with an AUC value of 1. Therefore, the AUC value can be derived by dividing the POC value by the POC cycle value per AU (eg, POC_cycle_au).

[345] In an embodiment, the per-AU POC cycle value (eg, poc_cyle_au) may be obtained from information located, for example, in the VPS that identifies the number of spatial layers or SNR layers in the encoded video sequence. Although derivation as described above can save a few bits in the VPS and therefore can improve encoding efficiency, it may be advantageous to explicitly encode the per-AU POC cycle value (e.g. poc_cycle_au) in an appropriate high-level syntax structure hierarchically below the VPS to minimize the value of the POC cycle per AU (for example, POC_cycle_au) for a given small part of the bitstream (for example, an image). This optimization can save more bits than can be stored by the acquisition process described above because POC values (and/or syntax element values indirectly referencing POC) can be encoded in low-level syntactic structures.

[346] In the embodiment of FIG. 20 shows example syntax tables for signaling the vps_poc_cycle_au syntax element in a VPS (or SPS), which specifies the poc_cycle_au used for all pictures/slices in the encoded video sequence, and the slice_poc_cycle_au syntax element, which specifies the poc_cycle_au of the current slice in the slice header.

[347] If the POC value increases uniformly for each AU, the vps_contant_poc_cycle_per_au syntax element in the VPS is set to 1, and the vps_poc_cycle_au syntax element is signaled in the VPS. Thus, the slice_poc_cycle_au syntax element is not explicitly signaled, and the AUC value for each AU can be calculated by dividing the POC value by the vps_poc_cycle_au syntax element.

[348] If the POC value does not increase uniformly across the AU, the vps_contant_poc_cycle_per_au syntax element in the VPS is set to 0. Thus, the vps_access_unit_cnt syntax element is not signaled, while the slice_access_unit_cnt syntax element is signaled in the slice header for each slice or image. Each slice or image can have a different slice_access_unit_cnt value. The AUC value for each AU can be calculated by dividing the POC value by the slice_POC_cycle_au syntax element.

[349] In FIG. 21 is a flowchart illustrating the corresponding workflow described above. The workflow starts from (S2101) and proceeds to (S2110).

[350] In (S2110), the VPS or SPS can be analyzed, and whether the POC cycle per AU is constant or not can be determined.

[351] At (S2120), it can be determined whether the POC cycle per AU is constant in the encoded video sequence. If the POC cycle per AU is determined to be constant in the encoded video sequence, the workflow proceeds to (S2130). Otherwise, the workflow proceeds to (S2140).

[352] In (S2130), the access unit counter value can be calculated from the sequence level poccycleau value and the POC value. The workflow then proceeds to (S2150).

[353] In (S2140), the value of the access unit counter can be calculated from the image level pos_cycle_au value and the POC value. The workflow then proceeds to (S2150).

[354] In (S2150), the VPS or SPS can be analyzed, and it can be determined whether the POC cycle per AU is constant.

[355] In an embodiment, even though the POC value of an image, slice, or tile may be different, an image, slice, or tile corresponding to AUs with the same AUC value may be associated with the same decoding or output timing. Therefore, without any dependency on inter-parsing/decoding between images, slices or tiles in the same AU, all or a subset of images, slices or tiles associated with the same AU can be decoded in parallel and can be released at the same point in time.

[356] In an embodiment, even though the POC value of an image, slice, or tile may be different, an image, slice, or tile corresponding to an AU with the same AUC value may be associated with the same arrangement/display time. When the layout time is contained in the container format, even if the images correspond to different AUs, as long as the images have the same layout time, the images can be displayed at the same point in time.

[357] In an embodiment, each image, slice or tile may have the same temporal identifier (eg, temporal_id) in the same AU. All or a subset of images, slices, or tiles corresponding to a point in time can be associated with the same time sublayer. In an embodiment, each image, slice, or tile may have the same or different spatial layer identifiers (eg, layer_id) in the same AU. All or a subset of images, slices, or tiles corresponding to a point in time can be associated with the same or a different spatial layer.

[358] In FIG. 22 shows an example of a video sequence structure with a combination of temporal_id, layer_id, POC and AUC values with adaptive resolution change. In the example shown in FIG. 22, an image, slice or tile in a first AU with AUC =0 may have temporal_id =0 and layer_id =0 or 1, while an image, slice or tile in a second AU with AUC =1 may have temporal_id =1 and layer_id =0 or 1 respectively. The POC value is increased by 1 for each image, regardless of the temporal_id and layer_id values. Referring to FIG. 22, the value of poc_cycle_au may be equal to 2. In the example, the value of poc_cycle_au may preferably be set equal to the number of layers (spatial scalability). In this example, the POC value is increased by 2 and the AUC value is increased by 1.

[359] In the above embodiments, all or a subset of the inter-frame or inter-layer prediction structure and reference picture indication may be supported using reference picture set (RPS) signaling in video standards such as HEVC or reference picture list (RPL) signaling. In RPS or RPL, the selected reference picture may be indicated by signaling a POC value or a POC delta value between the current picture and the selected reference picture. In various examples, RPS and RPL may be used to indicate an inter-picture or inter-layer prediction structure without changing the signaling, and with the following restrictions. If the temporal_id value of the reference picture is greater than the temporal_id value of the current picture, the current picture may not use the reference picture for motion compensation or other predictions. If the layer_id value of the reference image is greater than the layer_id value of the current image, the current image may not use the reference image for motion compensation or other predictions.

[360] In an embodiment, motion vector scaling based on the POC difference for temporal motion vector prediction may be disabled for multiple images in the AU. Therefore, although each image may have a different POC value in the AU, the motion vector cannot be scaled and used to temporally predict the motion vector in the AU, since a reference image with a different POC in the same AU can be considered as a reference image having the same time instant . Therefore, the motion vector scaling function may return 1 when the reference image belongs to the AU associated with the current image.

[361] In an embodiment, motion vector scaling based on the POC difference for temporal motion vector prediction can be optionally disabled for multiple images when the spatial resolution of the reference image is different from the spatial resolution of the current image. When motion vector scaling is enabled, the motion vector is scaled based on both the POC difference and the spatial resolution ratio between the current image and the reference image.

[362] In an embodiment, the motion vector may be scaled based on the AUC difference instead of the POC difference to temporally predict the motion vector, especially when poc_cycle_au has a non-uniform value (eg, when vps_contant_poc_cycle_per_au =0). Otherwise (eg, when vps_contant_poc_cycle_per_au =1), scaling the motion vector based on the AUC difference may be identical to scaling the motion vector based on the POC difference.

[363] In an embodiment, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU (with the same AUC value) with the current image is not scaled based on the AUC difference and is used to predict the motion vector without scaling or with scaling based on the spatial resolution relationship between the current image and the reference image.

[364] In an embodiment, the AUC value is used to identify the boundary of the AU and is used for a hypothetical reference decoder (HRD) operation that may require input and output synchronization to the AU granularity. In various examples, the decoded image with the highest level in the AU may be output for display. The AUC value and layer_id value can be used to identify the output image.

[365] In an embodiment, an image may comprise one or more sub-images. Each sub-image can cover a local area or the entire image area. The area supported by a sub-image may or may not overlap with the area supported by another sub-image. An area composed of one or more sub-images may or may not cover the entire image area. If the image includes sub-images, the area supported by the sub-image may be identical to the area supported by the image.

[366] In an embodiment, the sub-image may be encoded with a coding method similar to the encoding method used for the encoded image. The sub-image may be independently encoded or may be encoded depending on another sub-image or encoded image. A sub-image may or may not have any parsing dependency on another sub-image or encoded image.

[367] In an embodiment, the encoded sub-image may be contained in one or more layers. The encoded sub-image in a layer may have a different spatial resolution. The original sub-image may be spatially resampled (eg, upsampled or downsampled), encoded at different spatial resolution parameters, and contained in a bitstream corresponding to the layer.

[368] In an embodiment, the image fragment c (W, H) may be encoded and contained in the encoded bitstream corresponding to layer 0. W indicates the width of the sub-image and H indicates the height of the sub-image, respectively. An upsampled (or downsampled) subimage may be upsampled (or downsampled) from a portion of the original spatial resolution image. The upsampled (or downsampled) sub-image may have a spatial resolution (W*S _{w, k} , H*S _{h, k} ) and may be encoded and contained in a coded bitstream corresponding to level k. The parameters S _{w, k} and S _{h, k} may indicate the horizontal and vertical resampling ratios, respectively. If the values of S _w,k , S _{h, k} are greater than 1, the resampling is upsampling. If the values of S _{w, k} , S _{h, k} are less than 1, the resampling is downsampling.

[369] In an embodiment, an encoded sub-image in a layer may have a different visual quality than an encoded sub-image in another layer in the same sub-image or a different sub-image. For example, sub-image i in layer n is encoded with a quantization parameter (for example, Q _{i, n} ), while sub-image j in layer m is encoded with a quantization parameter (for example, Q _{J, m} .

[370] In an embodiment, an encoded sub-image in a layer can be decoded independently, without any dependence on parsing or decoding, from an encoded sub-image in another layer of the same local area. A sub-image layer that can be independently decoded without reference to another sub-image layer of the same local region is an independent sub-image layer. An encoded sub-image in an independent sub-image layer may or may not have a decoding or parsing dependency on a previously encoded sub-image in the same sub-image layer. The encoded sub-image may not have any dependence on the encoded image in another sub-image layer.

[371] In an embodiment, an encoded sub-image in a layer may be decoded dependently, with any parsing or decoding dependency, from an encoded sub-image in another layer of the same local area. A sub-image layer that can be dependently decoded with reference to another sub-image layer of the same local region is a dependent sub-image layer. An encoded subimage in a dependent subimage may reference an encoded subimage belonging to the same subimage, a previously encoded subimage in the same subimage layer, or both reference subimages.

[372] In an embodiment, the encoded sub-image contains one or more independent sub-image layers and one or more dependent sub-image layers. However, at least one independent sub-picture may be present for the encoded sub-picture. An independent sub-image layer may have a layer identifier (layer_id) value, which may be present in a NAL unit header or other high-level syntax structure, equal to 0. A sub-image layer with a layer_id equal to 0 is the base sub-image layer.

[373] In an embodiment, the image may include one or more foreground sub-images and one background sub-image. The area supported by the background subimage may be equal to the image area. The region supported by the foreground sub-image may overlap with the region supported by the background sub-image. The background sub-image may be a base sub-image layer, while the foreground sub-image may be a non-base sub-image layer (sub-image enhancement layer). One or more non-base layers of a sub-image may reference the same base layer for decoding. Each non-base subimage layer with layer_id equal to a can reference a non-base subimage layer with layer_id equal to b, where a is greater than b.

[374] In an embodiment, the image may include one or more foreground sub-images with or without one background sub-image. Each sub-image can have its own sub-image base layer and one or more non-base (enhancement) layers. Each subimage base layer may be referenced by one or more non-base subimage layers. Each non-base subimage layer with layer_id equal to a can reference a non-base subimage layer with layer_id equal to b, where a is greater than b.

[375] In an embodiment, the image may include one or more foreground sub-images with or without one background sub-image. Each encoded sub-image in a (base or non-base) sub-image layer may be referenced by one or more non-base layer sub-images that belong to the same sub-image, and one or more non-base layer sub-images that do not belong to the same sub-image.

[376] In an embodiment, the image may include one or more foreground sub-images with or without one background sub-image. The sub-image in layer a can be further divided into multiple sub-images in the same layer. One or more coded sub-pictures in layer b may reference a divided sub-picture in layer a.

[377] In an embodiment, a coded video sequence (CVS) may be a group of coded pictures. The CVS may include one or more coded sub-picture sequences (CSPS), where a CSPS may be a group of coded sub-pictures covering the same local image region. CSPS may have the same or a different temporal resolution as the encoded video sequence.

[378] In an embodiment, the CSPS may be encoded and contained in one or more layers. The CSPS may include one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS can reconstruct a sequence of sub-images corresponding to the same local region.

[379] In an embodiment, the number of CSPS layers corresponding to a CSPS may be identical to or different from the number of CSPS layers corresponding to another CSPS.

[380] In an embodiment, a CSPS layer may have a different temporal resolution (eg, frame rate) than another CSPS layer. The original (eg, uncompressed) subimage sequence may be temporally resampled (eg, upsampled or downsampled), encoded at different temporal resolution parameters, and contained in a bitstream corresponding to the layer.

[381] In an embodiment, a sub-image sequence at frame rate F may be encoded and contained in the encoded bitstream corresponding to layer 0, while a sub-image sequence is temporally upsampled (or down-sampled) from the original sub-image sequence at F*S _{t, k} may be encoded and contained in a coded bitstream corresponding to layer k, where S _t,k indicates the time sampling rate for layer k. If the value of S _t,k is greater than 1, the temporal resampling process corresponds to frame rate upconversion. If the value of S _t,k is less than 1, the temporal resampling process corresponds to frame rate down-conversion.

[382] In an embodiment where a sub-image with CSPS layer a is a reference to a sub-image with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a differs from the spatial resolution of CSPS layer b, the decoded pixels in CSPS layer a are resampled and are used for reference. The resampling process may require upsampling or downsampling filtering.

[383] In FIG. 23 shows an example video stream including a background CSPS video with a layer_id of 0 and a plurality of foreground CSPS layers. Although the encoded sub-image may include one or more CSPS layers, a background region that does not belong to any foreground CSPS layer may include a base layer. The base layer may contain a background region and a foreground region. The CSPS enhancement layer may contain a foreground region. The CSPS enhancement layer may have better visual quality than the base layer in the same area. The CSPS enhancement layer may reference the reconstructed pixels and motion vectors of the base layer corresponding to the same region.

[384] In an embodiment, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers corresponding to each sub-image are contained in a separate track in the video file.

[385] In embodiments, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers with the same layer_id are contained in a separate track. In the example, the track corresponding to layer k includes only the CSPS layers corresponding to layer k.

[386] In an embodiment, each CSPS layer of each sub-image is stored in a separate track. Each track may or may not have a parsing or decoding dependency on one or more other tracks.

[387] In an embodiment, each track may contain bitstreams corresponding to layer i to layer j of the CSPS layers of all or a subset of sub-images, where 0<i≤j≤k, and k is the highest CSPS layer.

[388] In embodiments, the image contains one or more types of associated media data, including a depth map, an alpha map, 3O geometry data, an occupancy map, etc. Such associated synchronized media data may be divided into one or multiple sub-data streams, each corresponding to one sub-image.

[389] In the embodiment of FIG. 24 illustrates an example of a video conference based on a method for creating multi-layer sub-images. The video stream contains one base layer video bitstream corresponding to a background image and one or more enhancement layer video bitstreams corresponding to foreground sub-images. Each video bitstream of the enhancement layer corresponds to a CSPS layer. The default display shows the image corresponding to the base layer. t contains one or more picture-in-picture (PIP) of the user. When a particular user is selected by the client control, the CSPS enhancement layer corresponding to the selected user is decoded and displayed with improved quality or spatial resolution.

[390] In FIG. 25 is a diagram of the above-described operation. The operation starts at (S2501) and proceeds to (S2510).

[391] At (S2510), the multi-layer video bitstream can be decoded.

[392] At (S2520), a background region and one or more foreground sub-images may be identified.

[393] At (S2530), it can be determined whether a specific sub-image area is selected. If it is determined that a specific sub-image area is to be selected, the operation proceeds to (S2540). Otherwise, operation proceeds to (S2550).

[394] At (S2540), the enhancement sub-picture can be decoded and displayed. The operation then proceeds to (S2599) and ends.

[395] At (S2550), the background region can be decoded and displayed. The operation then proceeds to (S2599) and ends.

[396] In an embodiment, a network intermediate unit (such as a router) may select a subset of layers to send to the user depending on its bandwidth. The image/sub-image organization can be used to adapt the bandwidth. For example, if the user does not have the bandwidth, the router separates the layers or selects some sub-images based on their importance or based on the settings used, and the operation can be performed dynamically to adapt to the bandwidth.

[397] In FIG. Figure 26 shows an example of using 360-degree video. When a 360 degree spherical image is projected onto a flat image, the 360 projection image can be divided into many sub-images as a base layer. The enhancement layer of a particular sub-image may be encoded and transmitted to the client. The decoder can decode both a base layer including all sub-images and an enhancement layer of a selected sub-image. When the current viewport is identical to the selected sub-image, the displayed image may have a higher quality with the decoded sub-image with an enhancement layer. Otherwise, the decoded image with the base layer may be displayed in poor quality.

[398] In an embodiment, any suitable layout information for display may be present in the file as additional information (such as an SEI message or metadata). One or more decoded sub-pictures may be moved and displayed depending on the signaled arrangement information. The layout information may be signaled by the streaming server or broadcaster, or may be retrieved by the network entity or cloud server, or may be determined by the user's individual settings.

[399] In an embodiment, when the input image is divided into one or more sub-regions (rectangular), each sub-region may be encoded as an independent layer. Each independent layer corresponding to a local area can have a unique layer_id value. For each independent layer, information about the size and location of the sub-image can be signaled. For example, image size (for example, image width and/or height), upper left corner offset information (for example, x_offset, y_offset). In fig. 27 shows an example of a divided sub-image layout, size and position information of its sub-images, and its corresponding image prediction structure. Layout information, including sub-image size(s) and sub-image position(s), may be signaled in a high-level syntactic structure such as parameter set(s), a slice or tile group header, or an SEI message.

[400] In an embodiment, each sub-image corresponding to an independent layer may have its own unique POC value within the AU. When a reference image among the images stored in the DPB is specified using syntax element(s) in an RPS or RPL structure, the POC value(s) of each sub-image corresponding to the layer may be used.

[401] In an embodiment, to indicate a prediction structure (eg, an inter-layer prediction structure), the layer_id may not be used, and a POC value (eg, a POC delta value) may be used.

[402] In an embodiment, a sub-image with a POC value of N corresponding to a layer (or local region) may or may not be used as a reference image of a sub-image with a POC value of N+K corresponding to the same layer (or the same local area). area) for motion compensated prediction. In various examples, the value of the number K may be equal to the maximum number of layers (eg, independent layers), which may be identical to the number of sub-regions.

[403] In the embodiment shown in FIG. 28 shows an expanded case of FIG. 27. When the input image is divided into multiple (eg four) sub-regions, each local region can be encoded with one or more layers. Referring to FIG. 28, the number of independent layers may be equal to the number of sub-regions, and one or more layers may correspond to a sub-region. Thus, each sub-region may be encoded by one or more independent layers and zero or more dependent layers.

[404] In the embodiment shown in FIG. 28, the input image can be divided into four sub-regions. The right upper sub-region may be encoded as two layers (eg, layer 1 and layer 4), and the right lower sub-region may be encoded as two layers (eg, layer 3 and layer 5). In FIG. 28, layer 4 may reference layer 1 for motion compensation prediction, and layer 5 may reference layer 3 for motion compensation.

[405] In an embodiment, intra-loop filtering (eg, deblocking filtering, adaptive intra-loop filtering, shapeshifting, two-way filtering, or any deep learning-based filtering) across a layer boundary can be (optionally) disabled.

[406] In an embodiment, motion compensated prediction or intra-block copying across a layer boundary can be (optionally) disabled.

[407] In an embodiment, boundary padding for motion-compensated prediction or intra-loop filtering at the sub-image boundary may be processed optionally. A flag indicating whether boundary padding is processed or not may be signaled in a high-level syntactic structure, such as a parameter set(s) (VPS, SPS, PPS, or APS), a slice or tile group header, or an SEI message.

[408] In an embodiment, the layout information of the sub-area(s) (or sub-image(s)) may be signaled to the VPS or SPS. Fig. 29A-29B show an example of syntax elements in VPS and SPS. Referring to FIG. 29A, a flag (eg, vps_sub_picture_dividing_flag) is signaled to the VPS. A flag (for example, vps_sub_picture_dividing_flag) can indicate whether the input pictures are divided into multiple sub-regions or not. When the value of the flag (for example, vps_sub_picture_dividing_flag) is 0, the input picture(s) in the encoded video sequence(s) corresponding to the current VPS cannot be divided into many sub-areas. Thus, the size of the input image may be equal to the size of the encoded image (eg, pic_width_in_luma_samples, pic_height_in_luma_samples) that is signaled in the SPS. When the value of a flag (eg vps_sub_picture_dividing_flag) is 1, the input image(s) can be divided into multiple sub-regions. So in this case the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled to the VPS. The values of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples can be equal to the width and height of the input image(s), respectively.

[409] In an embodiment, the values vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples cannot be used for decoding, but can be used for layout and display.

[410] In an embodiment, when the value of a flag (eg, vps_sub_picture_dividing_flag) is 1, the pic_offset_x and pic_offset_y syntax elements may be signaled in the SPS that corresponds to the specific layer(s). Thus, the encoded image size (eg, pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to a particular layer. The position (eg, pic_offset_x and pic_offset_y syntax elements) of the upper left corner of the sub-region may be signaled to the SPS.

[411] In an embodiment, position information (eg, pic_offset_x, pic_offset_y) of the upper left corner of the sub-region cannot be used for decoding, but can be used for layout and display.

[412] In an embodiment, layout information (eg, size and position) of all or a subset of subregions of the input image(s), as well as dependency information between layers, may be signaled in a parameter set or SEI message. In fig. 30 shows an example of syntax elements for specifying layout information of sub-regions, dependencies between layers, and a relationship between a sub-region and one or more layers. Referring to FIG. 30, the num_sub_region syntax element may indicate the number of (rectangular) subregions in the current encoded video sequence. The num_layers syntax element can indicate the number of layers in the current encoded video sequence. The value of num_layers can be equal to or greater than the value of num_sub_region. When any subregion is encoded as a single layer, the value of num_layers can be equal to the value of num_sub_region. When one or more subregions are encoded as multiple layers, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] can indicate the dependency of the jth layer on the ith layer. The num_layers_for_region[i] syntax element can specify the number of layers associated with the i-th subregion. The syntax element sub_region_layer_id[i][j] can specify the layer_id of the jth layer associated with the ith subregion. Sub_region_offset_x [i] and sub_region_offset_y [i] can indicate the horizontal and vertical position of the upper left corner of the i-th subregion, respectively. Sub_region_width[i] and sub_region_height[i] can specify the width and height of the i-th subregion, respectively.

[413] In an embodiment, one or more syntax elements that define a set of output layers to indicate one or more layers to be output with or without PTL information may be signaled in a high-level syntax structure, such as VPS, DPS, SPS, PPS APS or SEI message.

[414] In FIG. 31 syntax element num_output_layer_sets indicating the number of output layer sets (OLS) in the encoded video sequence referencing the VPS may be signaled to the VPS. For each set of output layers, a flag (eg, output_layer_flag) may be signaled for each of the output layers.

[415] In an embodiment, output_layer_flag[i] equal to 1 may indicate that the i-th layer is the output layer. A vps_output_layer_flag[i] value of 0 may indicate that the i-th layer is not an output layer.

[416] In an embodiment, one or more syntax elements that define PTL information for each set of output layers may be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message. Referring to FIG. 31, a num_profile_tile_level syntax element indicating the amount of profile tier level information on the OLS in a coded video sequence referring to the VPS may be signaled to the VPS. For each set of output layers, a set of syntax elements for PTL information or an index indicating specific PTL information among entries in the PTL information may be signaled for each output layer.

[417] In an embodiment, profile_tier_level_idx[i][j] specifies the index in the profile_tier_level() syntax structure list in the VPS of the profile_tier_level() syntax structure that applies to the j-th layer of the i-th OLS.

[418] In the embodiment as shown in FIG. 32, the num_profile_tile_level and/or num_output_layer_sets syntax elements may be signaled when the number of maximum layers is greater than 1 (vps_max_layers_minus1 >0).

[419] In embodiments as shown in FIG. 32, a syntax element vps_output_layers_mode[i] indicating the output layer signaling mode for the i-th set of output layers may be present in the VPS.

[420] In the same embodiment, vps_output_layers_mode[i] equal to 0 indicates that only the topmost layer is output with the i-th set of output layers; vps_output_layer_mode[i] equal to 1 indicates that all layers are output with the i-th set of output layers; vps_output_layer_mode[i] equal to 2 indicates that the output layers are those with vps_output_layer_flag[i][j] equal to 1 with the i-th set of output layers. Other values may be reserved.

[421] In an embodiment, output_layer_flag[i][j] may or may not be signaled depending on the value of vps_output_layers_mode[i] for the i-th set of output layers.

[422] In the embodiment as shown in FIG. 32, the flag vps_ptl_signal_flag[i] may be present for the i-th set of output layers. Depending on the value of vps_ptl_signal_flag[i], the PTL information for the i-th set of output layers may or may not be signaled.

[423] In the embodiment of FIG. 33, the number of subpics (eg, max_subpics_minus1) in the current CVS may be signaled in a high-level syntax structure, such as a VPS, DPS, SPS, PPS, APS, or SEI message.

[424] In the embodiment as shown in FIG. 33, a subpicture ID (eg, sub_pic_id[i]) for the i-th subpicture may be signaled when the number of subpictures is greater than 1 (max_subpics_minus1 >0).

[425] In an embodiment, one or more syntax elements indicating a sub-image identifier belonging to each layer of each set of output layers may be signaled to the VPS. Referring to FIG. 34, sub_pic_id_layer[i][j][k], which indicates the k-e sub-image present in the j-th layer of the i-th set of output layers. Using the above information, the decoder can determine which sub-image can be decoded and output for each layer of a particular set of output layers.

[426] In an embodiment, a picture header (PH) is a syntax structure containing syntax elements that apply to all slices of an encoded picture. The slice header may be part of an encoded slice containing data elements related to all tiles or CTU lines in the tile represented in the slice. A picture unit (PU) is a set of NAL units that are related to each other according to a given classification rule, are sequential in decoding order, and contain, for example, exactly one encoded picture. The PU may contain a picture header (PH) and one or more NAL VCL units constituting the encoded picture.

[427] In an embodiment, the SPS (RBSP) may be made available to the decoding process before it is referenced by the SPS, included in at least one AU with a TemporalId of 0, or provided through external means.

[428] In an embodiment, the SPS (RBSP) may be available to the decoding process before the SPC is referenced, it is included in at least one AU with a TemporalId of 0 in a CVS that contains one or more PPSs, referencing SPS, or provided through external means.

[429] In an embodiment, the SPS (RBSP) may be available to the decoding process before the SPS is referenced by one or more PPSs, it is included in at least one PU with a nuh_layer_id equal to the smallest nuh_layer_id value of the NAL PPS units that refer to an SPS NAL unit in a CVS that contains one or more PPS referencing the SPS or is provided through external means.

[430] In an embodiment, the SPS (RBSP) may be available to the decoding process before the SPS is referenced by one or more PPSs, it is included in at least one PU with a TemporalId equal to 0 and nuh_layer_id equal to the smallest value nuh_layer_id of NAL PPS units that reference a NAL SPS unit or are provided through external means.

[431] In an embodiment, the SPS (RBSP) may be available to the decoding process before the SPS is referenced by one or more PPSs, it is included in at least one PU with a TemporalId equal to 0 and nuh_layer_id equal to the smallest value of nuh_layer_id NAL PPS units that reference a NAL SPS unit in a CVS that contains one or more PPS that reference an SPS or is provided through external means.

[432] In an embodiment, pps_seq_parameter_set_id indicates the sps_seq_parameter_set_id value for the referenced SPS. In the example, the value of pps_seq_parameter_set_id could be the same across all PPSs referenced by encoded images in CLVS.

[433] In an embodiment, all SPS NAL units with a particular sps_seq_parameter_set_id value in CVS may have the same content.

[434] In an embodiment, regardless of nuh_layer_id values, SPS NAL units can share the same sps_seq_parameter_set_id value space.

[435] In an embodiment, the nuh_layer_id value of the SPS NAL unit may be equal to the smallest nuh_layer_id value of the PPS NAL units that reference the SPS NAL unit.

[436] In an embodiment, when an SPS with nuh_layer_id equal to m is referenced by one or more PPS with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or ( direct or indirect) reference layer of a layer with nuh_layer_id equal to m.

[437] In an embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the NAL PPS unit, or provided through external means.

[438] In an embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced, it is included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS that contains one or more RNs (or encoded slice NAL units) referencing the PPS, or provided through external means.

[439] In an embodiment, a PPS (RBSP) may be available to the decoding process before the PPS is referenced by one or more PHs (or encoded slice NAL units), it is included in at least one PU with nuh_layer_id equal to the smallest the nuh_layer_id value of coded slice NAL units that reference a PPS NAL unit in a CVS that contains one or more PHs (or coded slice NAL units) that reference a PPS, or is provided through external means.

[440] In an embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced by one or more PHs (or coded slice NAL units) included in at least one PU with a TemporalId equal to the TemporalId of the NAL unit PPS and nuh_layer_id equal to the smallest value of the nuh_layer_id of coded slice NAL units that reference a PPS NAL unit in a CVS that contains one or more PHs (or coded slice NAL units) that reference a PPS, or is provided through external means.

[441] In an embodiment, the ph_pic_parameter_set_id in the PH indicates the pps_pic_parameter_set_id value for the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by encoded images in CLVS.

[442] In an embodiment, all NAL PPS units with a particular pps_pic_parameter_set_id value in the PU may have the same content.

[443] In an embodiment, regardless of nuh_layer_id values, NAL PPS blocks can share the same pps_pic_parameter_set_id value space.

[444] In an embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the smallest nuh_layer_id value of the encoded slice NAL units that reference the NAL unit. The NAL unit refers to the PPS NAL unit.

[445] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more NAL units of an encoded slice with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n , or the (direct or indirect) reference layer of a layer with nuh_layer_id equal to m.

[446] In an embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit, or provided through external means.

[447] In an embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced, it is included in at least one AU with a TemporalId equal to the TemporalId of the PPS NAL unit in the CVS that contains one or more RNs (or encoded slice NAL units) referencing the PPS, or provided through external means.

[448] In an embodiment, a PPS (RBSP) may be available to the decoding process before the PPS is referenced by one or more PHs (or encoded slice NAL units), it is included in at least one PU with nuh_layer_id equal to the smallest the nuh_layer_id value of coded slice NAL units that reference a PPS NAL unit in a CVS that contains one or more PHs (or coded slice NAL units) that reference a PPS, or is provided through external means.

[449] In an embodiment, the PPS (RBSP) may be available to the decoding process before the PPS is referenced by one or more PHs (or encoded slice NAL units) included in at least one PU with a TemporalId equal to the TemporalId of the NAL unit PPS and nuh_layer_id equal to the smallest value of the nuh_layer_id of coded slice NAL units that reference a PPS NAL unit in a CVS that contains one or more PHs (or coded slice NAL units) that reference a PPS, or is provided through external means.

[450] In an embodiment, the ph_pic_parameter_set_id in the PH indicates the pps_pic_parameter_set_id value for the referenced PPS in use. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by encoded images in CLVS.

[451] In an embodiment, all NAL PPS units with a particular pps_pic_parameter_set_id value in the PU may have the same content.

[452] In an embodiment, regardless of nuh_layer_id values, NAL PPS blocks can share the same pps_pic_parameter_set_id value space.

[453] In an embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the smallest nuh_layer_id value of the encoded slice NAL units that reference the PPS NAL unit.

[454] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more NAL units of an encoded slice with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n , or the (direct or indirect) reference layer of a layer with nuh_layer_id equal to m.

[455] An output layer indicates a layer from an output layer set that is output. An output layer set (OLS) indicates a layer set including a specified layer set, where one or more layers in the layer set are indicated as output layers. The OLS Level Index is the level index in OLS. The OLS layer index may point to a list of layers in the OLS.

[456] The sublayer indicates a temporary scalable layer of a temporary scalable bitstream comprising VCL NAL units with a particular TemporalId variable value and associated non-VCL NAL units. The sublayer representation specifies a subset of the bitstream that includes the NAL units of a particular sublayer and lower sublayers.

[457] The RBSP VPS may be available to the decoding process prior to reference to the RBSP VPS, included in at least one AU with a TemporalId of 0, or provided through external means. All VPS NAL units with a specific vps_video_parameter_set_id value in CVS can have the same content.

[458] vps_video_parameter_set_id provides a VPS identifier for reference by other syntax elements. The value of vps_video_parameter_set_id can be greater than 0.

[459] vps_max_layers_minus1 plus 1 specifies the maximum number of layers allowed in each CVS referencing the VPS.

[460] vps_max_sublayers_minus1 plus 1 specifies the maximum number of temporary sublayers that can be present in a layer in each CVS referencing the VPS. The value of vps_max_sublayers_minus1 can be in the range from 0 to 6 inclusive.

[461] In the example, a flag (eg vps_all_layers_same_num_sublayers_flag) equal to 1 indicates that the number of temporary sublayers is the same for all layers in each CVS referencing the VPS. A flag (e.g. vps_all_layers_same_num_sublayers_flag) equal to 0 indicates that the layers in each CVS associated with a VPS may or may not have the same number of temporary sublayers. If absent, the value of the flag (for example, vps_all_layers_same_num_sublayers_flag) is set to 1.

[462] A flag (eg vps_all_independent_layers_flag) equal to 1 indicates that all layers in CVS are independently encoded without using inter-layer prediction. A flag (eg vps_all_independent_layers_flag) equal to 0 indicates that one or more layers in CVS can use inter-layer prediction. If it is missing, the value of the flag (for example, vps_all_independent_layers_flag) is output as 1.

[463] vps_layer_id [i] can specify the nuh_layer_id value of the i-th layer. For any two non-negative integer type values, the value of vps_layer_id[m] can be less than vps_layer_id[n] when m is less than n.

[464] vps_independent_layer_flag [i] equal to 1 may indicate that layer index i does not use cross-layer prediction, vps_independent_layer_flag [i] equal to 0 may indicate that layer index i can use cross-layer prediction, and syntax elements vps_direct_ref_layer_flag [464] i] [j] for j in the range 0 to i-1 inclusive are present in the VPS. If absent, the value of vps_independent_layer_flag [i] is output equal to 1.

[465] vps_direct_ref_layer_flag [i] [j] equal to 0 may indicate that layer index j is not the direct reference layer for layer index i. In this case, vps_direct_ref_layer_flag [i] [j] equal to 1 may indicate that the layer with index j is the direct reference layer for the layer with index i. When vps_direct_ref_layer_flag[i][j] is not present for i and j in the range 0 to vps_max_layers_minus1 inclusive, it may be output equal to 0. When vps_independent_layer_flag[i] is 0, there may be at least one value of j in the range 0 to i-1 inclusive, so the value of vps_direct_ref_layer_flag[i][j] is 1.

[466] The variables NumDirectRefLayers[i], DirectRefLayerldx[i][d], NumRefLayers[i], RefLayerldx[i][r], and LayerUsedAsRefLayerFlag[j] can be obtained as follows:

[467] The variable GeneralLayerldx [i], which defines the layer index for a layer with nuh_layer_id equal to vps_layer_id [i], can be obtained as follows: for (i=0; i<=vps_max_layers_minus1; i++) GeneralLayerldx [vps_layer_id [i]]= i

[468] In the example, for any two different values of i and j, both are in the range 0 to vps_max_layers_minus1 inclusive, when a flag (e.g., dependencyFlag[i][j]) is 1. The bitstream matching requirement could be that the values of chroma_format_idc and bit_depth_minus8 that apply to the i-th layer can be equal to the values of chroma_format_idc and bit_depth_minus8, respectively, that apply to the j-th layer.

[469] max_tid_ref_present_flag [i] equal to 1 may indicate that the max_tid_il_ref_pics_plus1 [i] syntax element is present. However, max_tid_ref_present_flag [i] equal to 0 may indicate that the syntax element max_tid_il_ref_pics_plus1 [i] is missing.

[470] max_tid_il_ref_pics_plus1 [i] equal to 0 may indicate that inter-layer prediction is not used by non-IRAP images of the i-th layer, max_tid_il_ref_pics_plus1 [i] greater than 0 may indicate that for decoding images of the i-th layer, no image with TemporalId is greater max_tid_il_ref_pics_plus1 [i] - 1 is not used as ILRP. If absent, the value of max_tid_il_ref_pics_plus1 [i] is taken to be 7.

[471] An each_layer_is_an_ols_flag value of 1 can indicate that each OLS contains only one layer, and each layer itself in a CVS referencing a VPS is an OLS with a single layer included, which is the only output layer. An each_layer_is_an_ols_flag value of 0 may indicate that an OLS may contain more than one layer. If vps_max_layers_minus1 is 0, the value of each_layer_is_an_ols_flag is set to 1. Otherwise, when vps_all_independent_layers_flag is 0, the value of each_layer_is_an_ols_flag is set to 0.

[472] ols_mode_idc equal to 0 may indicate that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1 +1, where the i-th OLS includes layers with layer indices from 0 to i inclusive, and for each OLS only the layer itself is output high level in OLS.

[473] ols_mode_idc equal to 1 may indicate that the total number of OLS specified by the VPS is vps_max_layers_minus1 +1, the i-th OLS includes layers with layer indices from 0 to i inclusive. For each OLS, all OLS layers are output.

[474] An ols_mode_idc value of 2 may indicate that the total number of OLSs indicated by the VPS are explicitly signaled, and for each OLS, the output layers are explicitly signaled and the other layers are layers that are direct or indirect reference layers of the OLS output layers.

[475] The value of ols_mode_idc can range from 0 to 2, inclusive. The ols_mode_idc value 3 may be reserved for certain video encoding technologies or standards, such as ITU-T or ISO/IEC.

[476] When a flag (for example, vps_all_independent_layers_flag) is 1 and a flag (for example, each_layer_is_an_ols_flag) is 0, the value of ols_mode_idc is output as 2.

[477] num_output_layer_sets_minus1 plus 1 can indicate the total number of OLS specified by the VPS when ols_mode_idc is 2.

[478] The TotalNumOlss variable indicating the total number of OLSs specified by the VPS can be output as follows:

[479] ols_output_layer_flag [i] [j] equal to 1 may indicate that the layer with nuh_layer_id equal to vps_layer_id [j] is the output layer of the i-th OLS when ols_mode_idc is equal to 2. ols_output_layer_flag [i] [j] is equal to 0 can indicate that the layer with nuh_layer_id equal to vps_layer_id[j] is not the output layer of the i-th OLS when ols_mode_idc is equal to 2.

[480] Variable NumOutputLayerslnOls [i], indicating the number of output layers in the i-th OLS, variable NumSubLayerslnLayerlnOLS [i] [j], indicating the number of sublayers in the j-th layer in the i-th OLS, variable OutputLayerldlnOls [i] [j] , which determines the nuh_layer_id value of the j-th output layer in the i-th OLS, and the variable LayerUsedAsOutputLayerFlag [k], which determines whether the k-th layer is used as an output layer in at least one OLS, can be obtained as follows:

[481] For each value of i in the range 0 to vps_max_layers_minus1 inclusive, the values LayerUsedAsRefLayerFlag[i] and LayerUsedAsOutputLayerFlag[i] cannot be equal to 0 at the same time. Thus, there may be no layer that is neither an output layer of at least one OLS nor a direct reference layer of any other layer.

[482] For each OLS there may be at least one layer that is an output layer. Thus, for any value of i in the range 0 to TotalNumOlss - 1 inclusive, the value of NumOutputLayerslnOls[i] can be greater than or equal to 1.

[483] You can get the NumLayerslnOls[i] variable that defines the number of layers in the i-th OLS, and the LayerldlnOls[i][j] variable that defines the nuh_layer_id value of the j-th layer in the i-th OLS as follows:

[484] The OlsLayerldx[i][j] variable defining the OLS layer index for a layer with nuh_layer_id equal to LayerldlnOls[i][j] can be obtained as follows:

[485] The lowest layer in each OLS may be an independent layer. Thus, for each i in the range from 0 to TotalNumOlss - 1 inclusive, the value of vps_independent_layer_flag [GeneralLayerldx [LayerldlnOls [i] [0]]] can be equal to 1.

[486] Each layer can be included in at least one OLS specified by the VPS. Thus, for each layer with a particular nuh_layer_id nuhLayerld value equal to one of vps_layer_id[k] for k in the range 0 to vps_max_layers_minus1 inclusive, there can be at least one pair of values i and j, where i is in the range 0 to TotalNumOlss - 1 inclusive, and j is in the range NumLayerslnOls[i] - 1 inclusive, so the value of LayerldlnOls[i][j] is nuhLayerld.

[487] In an embodiment, the decoding process for the current CurrPic picture works as follows:

- PictureOutputFlag is set as follows:

- If one of the following conditions is true, PictureOutputFlag is set to 0:

- the current image is a RASL image, and the NoOutputBeforeRecoveryFlag value of the associated IRAP image is 1.

- gdr_enabled_flag is equal to 1 and the current image is a GDR image with NoOutputBeforeRecoveryFlag equal to 1.

- the gdr_enabled_flag syntax element is equal to 1, the current image is associated with a GDR image with NoOutputBeforeRecoveryFlag equal to 1, and the PicOrderCntVal of the current image is less than the RpPicOrderCntVal of the associated GDR image;

- sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0, and the current AU contains an image picA that satisfies all of the following conditions:

- PicA has a PictureOutputFlag equal to 1.

- PicA nuh_layer_id has a nuhLid greater than that of the current image.

- PicA belongs to the OLS output layer (i.e. OutputLayerldlnOls[TargetOlsldx][0] is equal to nuhLid).

- sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 2 and ols_output_layer_flag [TargetOlsldx] [GeneralLayerldx [nuh_layer_id]] is equal to 0.

- Otherwise, PictureOutputFlag is set to pic_output_flag.

[488] After all slices of the current image are decoded, the currently decoded image is marked as "used for short-term reference" and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as "used for short-term reference".

[489] In embodiments where each layer is a set of output layers, PictureOutputFlag is set to pic_output_flag, regardless of the value of ols_mode_idc.

[490] In an embodiment, PictureOutputFlag is set to 0 when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is 0, ols_mode_idc is 0, and the current AU contains an image picA that satisfies all of the following conditions: PicA has a PictureOutputFlag equal to 1, PicA has a nuh_layer_id nuhLid greater than the current image, and PicA belongs to the OLS output layer (i.e. OutputLayerldlnOls[TargetOlsldx][0] is equal to nuhLid).

[491] In an embodiment, PictureOutputFlag is set to 0 when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is 0, ols_mode_idc is 2, and ols_output_layer_flag [TargetOlsldx][GeneralLayerldx[nuh_layer_id]] is 0.

[492] In FIG. Figure 35 shows an example of the syntax structure of an RBSP VPS. The syntax structure shown in FIG. 35 is similar to the syntax structure shown in FIG. 34, therefore detailed descriptions are omitted for brevity.

[493] The techniques described above (eg, techniques for signaling restriction flags, adaptive resolution parameters, and/or the like) may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, in FIG. 36 illustrates a computer system (3600) suitable for implementing certain embodiments of the disclosed subject matter.

[494] Computer software may be encoded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed directly or through interpretation, execution of microcode, and the like, one or more computer central processing units (CPUs), graphics processing units (GPUs), and the like.

[495] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[496] The components shown in FIG. 36 for a computer system (3600) are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of computer software implementing embodiments of the present disclosure. The component configuration should also not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary computer system embodiment (3600).

[497] The computer system (3600) may include certain human-machine interface input devices. Such a human-machine interface input device may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, control glove movement), audio input (e.g., voice, hand clapping), visual input (for example: gestures), olfactory input (not shown). Human-machine interface devices can also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, ambient sound), images (e.g. scanned images, photographic images captured from a camera for still images), video (e.g. 2D video, 3D video, including stereoscopic video).

[498] Human-machine interface input devices may include one or more of the following (only one of each is shown): keyboard (3601), mouse (3602), trackpad (3603), touch screen (3610), control glove (not shown), joystick (3605), microphone (3606), scanner (3607), camera (3608).

[499] The computer system (3600) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such HMI output devices may include haptic output devices (e.g., haptic feedback from a touch screen (3610), a control glove (not shown), or a joystick (3605), but there may also be haptic feedback devices that do not serve as input devices), audio output devices (for example: speakers (3609), headphones (not shown)), image output devices (for example, screens (3610), including CRT screens, LCD screens, plasma screens, OLED -screens, each with or without touchscreen input capabilities, each with or without haptic feedback capability - some of which may be capable of providing two-dimensional visual output or more than three-dimensional output through means such as stereographic output ; virtual reality glasses (not pictured), holographic displays and smoke tanks (not pictured)) and printers (not pictured).

[500] The computer system (3600) may also include human accessible storage devices and associated media, such as optical media, including CD/DVD ROM/RW (3620) with CD/DVD or similar media (3621), flash drive (3622), removable hard drive or solid state drive (3623), previously developed magnetic media such as tape and floppy disk (not pictured), specialized ROM/ASIC/PLD based devices such as dongles (not pictured) and so on.

[501] Those skilled in the art will also understand that the term "machine readable medium" as used in connection with the subject matter disclosed herein does not cover transmission media, carrier waves, or other time signals.

[502] The computer system (3600) may also include an interface to one or more communications networks. Networks can be, for example, wireless, wired, optical. In addition, networks can be local, global, metropolitan, automotive and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired television or wireless wide area digital networks including cable television, satellite television and terrestrial TV broadcasting, automotive and industrial equipment, including CAN bus, etc. Some networks typically require external network interface adapters that are connected to certain general purpose data ports or peripheral buses (3649) (for example, a computer system's USB ports (3600)); others are typically integrated into the core of the computer system (3600) by connecting to the system bus as described below (for example, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (3600) can communicate with other entities. Such communication may be unidirectional, receive only (for example, broadcast television), unidirectional, send only (for example, CAN bus to certain CAN bus devices), or bidirectional, such as for other computer systems using a local or wide area digital network . Each of these networks and network interfaces may use specific protocols and protocol stacks, as described above.

[503] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core (3640) of the computer system (3600).

[504] The core (3640) may include one or more central processing units (CPUs) (3641), graphics processing units (GPUs) (3642), field programmable gate area (FPGA) processors (3643), hardware ( 3644), graphics accelerators (3650), etc. These devices, along with read-only memory (ROM) (3645), random access memory (3646), internal storage devices such as internal hard drives not accessible to the user, solid state drives, etc. (3647), can be connected via system bus (3648). On some computer systems, the system bus (3648) may be available as one or more physical connectors to allow expansion with additional CPUs, GPUs, etc. Peripherals can be connected either directly to the system bus (3648) of the core, or through a peripheral bus (3649). For example, a screen (3610) can be connected to a graphics adapter (3650). Peripheral bus architectures include PCI, USB, etc.

[505] The CPU (3641), GPU (3642), FGPA (3643), and accelerators (3644) may execute certain instructions that, when combined, may constitute the aforementioned computer code. This computer code can be stored in ROM (3645) or RAM (3646). Transient data can also be stored in RAM (3646), while persistent data can be stored in, for example, an internal mass storage device (3647). Fast storage and retrieval to any of the storage devices can be enabled through the use of cache memory, which can be tightly coupled to one or more of the CPU (3641), GPU (3642), mass storage (3647), ROM (3645), RAM (3646) and the like.

[506] A computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specifically designed and constructed for the purposes of the present invention, or they may be well known and available to those skilled in the computer software art.

[507] By way of example, and not limitation, a computer system having an architecture (3600) and, in particular, a core (3640), may provide functionality through processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.). etc.) executing software embodied on one or more tangible machine-readable media. Such computer-readable media may be media associated with user-accessible mass storage, as described above, as well as certain kernel storage (3640) that is non-transitory in nature, such as internal kernel mass storage (3647), or ROM (3645). Software implementing various embodiments of the present invention may be stored in such devices and executed by the kernel (3640). The computer readable medium may include one or more storage devices or chips according to specific needs. Software may cause the core (3640) and, in particular, processors therein (including CPUs, GPUs, FPGAs, and the like) to perform certain processes or certain portions of specific processes described herein, including defining data structures stored in RAM (3646) and modifying such data structures according to processes defined by the software. In addition or alternatively, a computer system may provide functionality as a result of logic built into hardware circuitry or otherwise embodied in circuitry (e.g., accelerator (3644)), which may operate instead of or in conjunction with software to perform certain processes or individual parts of the specific processes described in this document. A reference to software may include logic and vice versa where applicable. Reference to a computer-readable medium may include circuitry (eg, an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both, where applicable. The present invention includes any suitable combination of hardware and software.

Appendix A: Abbreviations

JEM: Joint Research Model

VVC: Universal Video Coding

BMS: test suite

MV: motion vector

HEVC: High Efficiency Video Coding

SEI: additional extended information

VUI: Video property information

GOPs: Groups of Images

TUs: conversion units,

PUs: Prediction Units

CTUs: Coding Tree Units

CTBs: Coding Tree Blocks

PBs: Prediction Blocks

HRD: Hypothetical Reference Decoder

SNR: Signal to Noise Ratio

CPUs: Central Processing Units

GPUs (GPU): graphic processing units

CRT (Cathode Ray Tube): Cathode Ray Tube

LCD: liquid crystal display

OLED: Organic Light Emitting Diode

CD: CD

DVD: Digital Video Disc

ROM (ROM): read-only memory

RAM (random access memory): random access memory

ASIC: Application Specific Integrated Circuit

PLD: Programmable Logic Device

LAN: local area network

GSM: Global System for Mobile Communications

LTE: long-term development

CANBus (CAN bus): controller local network bus

USB: Universal Serial Bus

PCI: Peripheral Component Interconnect

FPGA: Field Programmable Gate Arrays

SSD: Solid State Drive

IC: integrated circuit

CU: coding unit

[508] Although this description discloses several exemplary embodiments, there are variations, permutations, and various substitute equivalents that fall within the scope of the invention. Thus, it will be appreciated that those skilled in the art will be able to develop numerous systems and methods that, although not expressly shown or described herein, embody the principles of the invention and are thus within the spirit and scope thereof.

Claims (58)

1. A method for decoding video in a decoder, containing: obtaining a control flag for each of the at least one group of restriction information in the restriction information signaled in the encoded video bitstream, wherein each control flag indicates whether a group of restriction information is present from the at least one group of restriction information corresponding to the corresponding control flag, in restriction information, wherein the restriction information is for a subset of encoding blocks in the encoded video bitstream; determining, based on the control flag of the restriction information group from the at least one restriction information group, whether the restriction information group of the control flag is present in the restriction information, wherein the restriction information group of the control flag includes at least one restriction flag for a subset of encoding blocks; determining prediction information for a subset of encoding blocks based on whether a control flag constraint information group is present in the constraint information; And recovering a subset of coding blocks based on the prediction information. 2. The method according to claim 1, in which determining that the control flag restriction information group is present in the restriction information; And determining the prediction information includes determining prediction information for a subset of coding blocks based on at least one restriction flag in the control flag restriction information group. 3. The method according to claim 1, in which determine that the restriction information group of the control flag shall not be present in the restriction information. 4. The method of claim 1, wherein each control flag indicates that a group of restriction information corresponding to the corresponding control flag is not present in the restriction information, and there are no restriction flags in the restriction information. 5. The method of claim 1, wherein the constraint information is present in high-level syntax for at least one image, and the subset of encoding blocks includes encoding blocks in at least one image. 6. The method according to claim 1, in which one or more of at least one group of constraint information is present in the encoded video bitstream, a plurality of restriction flags in one or more of the at least one group of restriction information are signaled in accordance with a predetermined order, and determining the prediction information further includes performing run length decoding on the plurality of constraint flags; and determining prediction information for a subset of coding blocks based on the plurality of constraint flags. 7. The method according to claim 2, in which the at least one restriction flag in the control flag restriction information group includes a plurality of restriction flags signaled in accordance with a predetermined order, and determining the prediction information further includes performing run length decoding on the plurality of constraint flags. 8. The method according to claim 2, in which an offset of the restriction information and a length of the restriction information are present in the restriction information, indicating that at least one restriction flag in the restriction information group of the control flag is present in the restriction information. 9. The method according to claim 1, in which one or more of the at least one group of restriction information is present in the encoded video bitstream; for each of one or more of the at least one group of constraint information, an offset of the constraint information and a length of the constraint information are present in the constraint information indicating that at least one constraint flag is in the corresponding one of one or more of the at least one group of constraint information present in the restriction information; And the number of offsets of the constraint information is signaled in the encoded video bitstream. 10. The method according to claim 1, in which one or more of at least one group of constraint information is present in the encoded video bitstream, one or more syntax elements in the restriction information are byte aligned, wherein the one or more syntax elements include a control flag for each of the at least one group of restriction information and a plurality of restriction flags in one or more of the at least one group of restriction information . 11. A video decoding device comprising processing circuitry configured to: obtaining a control flag for each of the at least one group of restriction information in the restriction information signaled in the encoded video bitstream, each control flag indicating whether a group of restriction information is present from the at least one group of restriction information corresponding to the corresponding control flag, in restriction information, wherein the restriction information is for a subset of encoding blocks in the encoded video bitstream; determining, based on the control flag of the restriction information group from the at least one restriction information group, whether the restriction information group of the control flag is present in the restriction information, wherein the restriction information group of the control flag includes at least one restriction flag for a subset of encoding blocks; determining prediction information for a subset of encoding blocks based on whether a control flag constraint information group is present in the constraint information; And reconstructing a subset of coding blocks based on the prediction information. 12. The device according to claim 11, in which it is determined that the control flag restriction information group is present in the restriction information; And the processing circuitry is configured to determine prediction information for a subset of encoding blocks based on at least one restriction flag in the control flag restriction information group. 13. The device according to claim 11, in which it is determined that the restriction information group of the control flag shall not be present in the restriction information. 14. The apparatus of claim 11, wherein each control flag indicates that a group of restriction information corresponding to the corresponding control flag is not present in the restriction information, and there are no restriction flags in the restriction information. 15. The apparatus of claim 11, wherein the constraint information is present in high-level syntax for at least one image, and the subset of encoding blocks includes encoding blocks in at least one image. 16. The device according to claim 11, in which one or more of at least one group of constraint information is present in the encoded video bitstream, a plurality of restriction flags in one or more of the at least one group of restriction information are signaled in accordance with a predetermined order, and the processing circuit is configured for: performing run length decoding on the plurality of limit flags; and determining prediction information for a subset of coding blocks based on the plurality of constraint flags. 17. The device according to claim 12, in which the at least one restriction flag in the control flag restriction information group includes a plurality of restriction flags signaled in accordance with a predetermined order, and the processing circuitry is configured to perform run length decoding on the plurality of constraint flags. 18. The device according to claim 12, in which an offset of the restriction information and a length of the restriction information are present in the restriction information, indicating that at least one restriction flag in the restriction information group of the control flag is present in the restriction information. 19. The device according to claim 11, in which one or more of at least one group of constraint information is present in the encoded video bitstream, one or more syntax elements in the restriction information are byte aligned, wherein the one or more syntax elements include a control flag for each of the at least one group of restriction information and a plurality of restriction flags in one or more of the at least one group of restriction information . 20. A non-transitory computer-readable storage medium storing instructions executable by at least one processor to execute: obtaining a control flag for each of the at least one group of restriction information in the restriction information signaled in the encoded video bitstream, each control flag indicating whether a group of restriction information is present from the at least one group of restriction information corresponding to the corresponding control flag, in restriction information, wherein the restriction information is for a subset of encoding blocks in the encoded video bitstream; determining, based on the control flag of the restriction information group from the at least one restriction information group, whether the restriction information group of the control flag is present in the restriction information, wherein the restriction information group of the control flag includes at least one restriction flag for a subset of encoding blocks; determining prediction information for a subset of encoding blocks based on whether a control flag constraint information group is present in the constraint information; And reconstructing a subset of coding blocks based on the prediction information.