RU2777924C1 - Method for signaling a set of output levels with a subimage - Google Patents

Abstract

FIELD: video technology.

SUBSTANCE: invention relates to means for video coding. Video data are received. The high-level syntax of video data is analyzed. Then, determining whether the value of the syntactic element of the high-level syntax indicates the value of the image sequence number (ISN) for the video data access unit (AU). Assigning at least one of the many images, slices and tiles of video data to the access unit based on the value of the syntactic element. Determining whether the high-level syntax includes a flag indicating whether at least one of the images is divided into multiple subdomains. Setting the size of the input image of at least one of the images equal to the size of the coded image signaled in the sequence parameter set (SPS) of video data, in response to the determination that the high-level syntax includes a flag indicating that at least one of the images is not divided into multiple subdomains.

EFFECT: increase in coding efficiency.

16 cl, 25 dwg, 4 tbl

Description

LINK TO RELATED APPLICATION

[0001] The present invention claims priority in U.S. Provisional Application No. 62/904,338, filed September 23, 2019, and U.S. Ordinary Application No. 17/024,288, filed September 17, 2020, the entire scope of which is incorporated herein.

BACKGROUND OF THE INVENTION

1. The field of technology to which the invention belongs

[0002] Presented in this application, the invention relates to video coding and decoding, and more specifically to signaling the profile/tier/level information to support temporal/spatial scalability through division into sub-images.

2. Description of the prior art

[0003] Motion-compensated inter-picture prediction video encoding and decoding has been in use for decades. color samples. The image sequence may have a fixed or variable image rate (informally also referred to as frame rate), such as 60 images per second, or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (resolution of 1920x1080 luminance samples at 60Hz frame rate) requires about 1.5Gbps of bandwidth. An hour of such video requires more than 600 GB of storage.

[0004] One goal of video encoding and decoding may be to reduce redundancy in the input video signal by compressing. Compression can help to alleviate the aforementioned bandwidth or storage requirements, in some cases by two orders of magnitude or more. You can use both lossless and lossy compression, as well as a combination of both. Lossless compression refers to techniques for reconstructing an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the discrepancy between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used for video. The amount of distortion allowed depends on the application; for example, users of some custom streaming applications can tolerate higher distortion than users of broadcast TV applications. With an achievable compression ratio, a higher allowed/allowable distortion may result in a higher compression ratio.

[0005] The video encoder and decoder may use techniques from several broad categories including, for example, motion compensation, transformation, quantization, and entropy coding. Some of these methods will be presented below.

[0006] Traditionally, video encoders and decoders worked with a given image size, which in most cases was determined and remained constant for a coded video sequence (CVS), a group of pictures (GOP), or a similar time frame with a group of pictures. For example, in MPEG-2, system decisions change the horizontal resolution (and hence image size) depending on factors such as scene activity, but only for I images, that is, typically for a group of images. Apply resampling of reference pictures for different resolutions in the encoded video sequence, for example in ITU-T Rec. H.263, Appendix P. In this case, the image size does not change, only reference images are recalculated, which potentially affects only the elements of the applied image (in case of a decrease in resolution) or only on the elements of the processed scene (when the resolution is increased). Further, H.263, Annex Q allows the resampling of an individual macroblock by halving or halving (on each dimension). Again, the image size remains unchanged. The macroblock size is fixed in H.263 and therefore does not need to be signaled.

[0007] Changes in the size of the predicted images have become a common trend in modern video coding. For example, VP9 allows you to resample the reference image and change the resolution of the entire image. Similar suggestions have been made for VVC (e.g., in Gendry et al. "Adaptive Resolution Resizing (ARC) in VVC", Joint Video Team document JVET-M0135-v1, Jan 9-19, 2019, incorporated herein in its entirety) allow resampling entire reference images to obtain different (higher or lower) resolutions. That document suggested that different resolutions be encoded by a video sequence parameter set with reference to the frame-by-frame syntax elements in the picture parameter set.

SUMMARY OF THE INVENTION

[0008] The invention includes a method and apparatus with a memory for storing computer program code and a processor(s) for accessing the computer program code and for operating in accordance with the computer program code. The computer program code is represented by an acquisition code configured to receive video data by at least one processor, a string data parsing code for parsing the video data video parameter set (VPS) syntax by at least one processor, a determination code configured to be determined by at least one processor, whether the value of the syntax element of the VPS syntax points to the value of the image sequence number (POC) of the access unit (AU) of video data, and the setting code for assigning at least one of the plurality of images, slices and tiles of video data to the access unit based on the value of the syntax element.

[0009] According to illustrative embodiments, the value of the syntax element indicates the number of multiple consecutive images, slices and tiles of video data to be assigned to the access unit.

[0010] According to exemplary embodiments, the VPS syntax resides in the video data VPS and specifies the number of at least one type of additional video data layers.

[0011] According to illustrative embodiments, the determination code is configured to cause at least one processor to determine whether the VPS syntax includes a flag indicating whether the value of the POC increases uniformly by AU.

[0012] According to illustrative embodiments, the calculation code is configured to cause at least one processor, in response to determining that the VPS turns on a flag indicating uneven increase in the POC value per access unit, to calculate the access unit number (AUC), based on the POC value and the image level value of the video data.

[0013] According to illustrative embodiments, the calculation code is configured to cause at least one processor, in response to determining that the VPS turns on a flag indicating a uniform increase in the POC value per access unit, to calculate the access unit number (AUC), based on the POC value of the video sequence level value.

[0014] According to exemplary embodiments, the determination code is configured to cause at least one processor to determine whether the VPS syntax includes a flag indicating whether at least one of the images is divided into a plurality of sub-regions.

[0015] According to illustrative embodiments, the setup code is configured to cause at least one processor, in response to determining that the VPS syntax includes a flag indicating that at least one of the images is not divided into a plurality of sub-regions, to set the size of at least one input image equal to the encoded image size signaled in the sequence parameter set (SPS) of the video data.

[0016] According to illustrative embodiments, the determination code is configured to cause at least one processor, in response to determining that the VPS syntax includes a flag indicating the division of at least one of the images into a plurality of sub-areas, to determine whether it contains SPS syntax elements signaling offsets corresponding to the video data layer.

[0017] According to exemplary embodiments, offsets are represented by a width and height offset.

BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Additional features, nature and various advantages of the disclosed invention will be apparent from the following detailed description and the accompanying drawings, in which:

[0019] FIG. 1 is a schematic representation of a simplified block diagram of a communication system in accordance with embodiments.

[0020] FIG. 2 is a schematic diagram of a simplified block diagram of a communication system in accordance with embodiments.

[0021] FIG. 3 is a schematic diagram of a simplified block diagram of a decoder in accordance with embodiments.

[0022] FIG. 4 is a schematic diagram of a simplified block diagram of an encoder in accordance with embodiments.

[0023] FIG. 5A is a schematic representation of options for signaling ARC parameters in accordance with the related art.

[0024] FIG. 5B is a schematic representation of options for signaling ARC parameters in accordance with the related art.

[0025] FIG. 5C is a schematic representation of options for ARC parameter signaling in accordance with embodiments.

[0026] FIG. 5D is a schematic representation of options for ARC parameter signaling, in accordance with embodiments.

[0027] FIG. 5E is a schematic representation of options for ARC parameter signaling in accordance with embodiments.

[0028] FIG. 6 is an example of a syntax table according to embodiments.

[0029] FIG. 7 is a schematic representation of a computer system in accordance with embodiments.

[0030] FIG. 8 is an example of a prediction structure for adaptive resolution scalability.

[0031] FIG. 9 is an example of a syntax table according to embodiments.

[0032] FIG. 10 is a schematic diagram of a simplified flowchart of parsing and decoding a ROC cycle per access block and an access block number value, in accordance with embodiments.

[0033] FIG. 11 is a schematic diagram of a video bitstream structure with layered sub-images according to embodiments.

[0034] FIG. 12 is a schematic representation of a selected sub-image with enhanced resolution in accordance with embodiments.

[0035] FIG. 13 is a flowchart of a decoding and display process for a video bitstream with layered sub-images according to embodiments.

[0036] FIG. 14 is a schematic representation of a video display device 360 with an additional sub-image layer in accordance with embodiments.

[0037] FIG. 15 shows an example of sub-image arrangement data, their respective level, and an image prediction structure according to the embodiments.

[0038] FIG. 16 is an example of sub-image arrangement data, their respective layer and image prediction structure with local area spatial scalability modality according to embodiments.

[0039] FIG. 17 is an example of a syntax table for sub-picture arrangement data according to embodiments.

[0040] FIG. 18 is an example of an SEI message syntax table for sub-picture arrangement data according to embodiments.

[0041] FIG. 19 is an example of a syntax table for specifying output levels and profile/tier/level information for each set of output levels, in accordance with embodiments.

[0042] FIG. 20 is an example of a syntax table for specifying an output level mode for each set of output levels, in accordance with embodiments.

[0043] FIG. 21 is an example of a syntax table for indicating the current sub-image of each layer for each set of output layers, in accordance with embodiments.

DETAILED DESCRIPTION

[0044] The methods below can be used alone or combined in any order. Embodiments may be executed by processing circuitry (eg, one or more processors, as well as one or more integrated circuits). In one example, one or more processors execute a program stored on a computer-readable storage medium.

[0045] Recently, a method of aggregating a compressed area or extracting multiple semantically independent picture elements in a single video image has attracted attention. In particular, for example, in the context of 360° encoding or certain video surveillance applications, multiple semantically independent source images (for example, a six-cube 360° cube scene surface or incoming camera images in a multi-camera surveillance setup) may require separate adaptive resolution settings to handle different scenes at a given point in time. In other words, encoders at a given point in time may choose different resampling factors for different semantically independent images that make up the entire video surveillance scene or 360° scene. When assembled into a single picture, which in turn requires resampling of the reference picture, adaptive resolution coding signaling is available for portions of the encoded picture.

[0046] FIG. 1 shows a simplified block diagram of a communication system (100) according to an embodiment of the present invention. The system (100) may include at least two terminals (110, 120) connected by a network (150). In unidirectional data transmission, the first terminal (110) may encode the video data in place for transmission to another terminal (120) via the network (150). The second terminal (120) may receive the encoded video data of another terminal via the network (150), decode the encoded data, and display the recovered video data. One-way communication may be inherent in media service applications and the like.

[0047] FIG. 1 shows a second pair of terminals (130, 140) for bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bi-directional data transmission, each terminal (130, 140) may encode video data received locally for transmission to another terminal over the network (150). Each terminal device (130, 140) may also receive the encoded video data transmitted by the other terminal device, decode the encoded data, and display the recovered video data on a local display device.

[0048] FIG. 1 terminal devices (110, 120, 130, 140) can be illustrated as servers, personal computers and smart phones, but this does not limit the principles of the present invention. Embodiments of the present invention find application in laptop computers, tablet computers, media players and/or dedicated video conferencing equipment. Network (150) represents any number of networks that carry encoded video data between terminals (110, 120, 130, 140), including, for example, wired and/or wireless communication networks. The communication network (150) allows data to be exchanged in a circuit-switched and/or packet-switched mode. Exemplary networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For the purposes of this discussion, the architecture and topology of the network (150) may not be relevant to the present invention unless specifically noted below.

[0049] FIG. 2 shows, as an example of the application of the disclosed invention, the placement of a video encoder and a video decoder in a streaming environment. The disclosed invention may be equally applicable to other video processing applications, including, for example, video conferencing, digital television, storage of compressed video on digital media, including CD, DVD, memory card, etc., etc.

[0050] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a digital camera, creating, for example, an uncompressed video stream (202). The sample stream (202), shown in bold to highlight the large amount of data compared to the encoded video data, may be processed by an encoder (203) connected to a camera (201). The encoder (203) may include hardware, software, or a combination thereof for providing or implementing aspects of the disclosed invention, as described in more detail below. The encoded video data (204), shown with a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server (405) for future use. One or more streaming clients (206, 208) may access the streaming server (205) to retrieve copies (207, 209) of the encoded video data (204). The client (206) may include a video decoder (210) that decodes an incoming copy of the encoded video data (207) and creates an outgoing video stream (211) that can be rendered on a display (212) or other rendering device (not shown). In some streaming systems, the encoded video data (204, 207, 209) may be encoded according to one or another video coding/compression standard. Examples of these standards include ITU-T Recommendation H.265. A video coding standard is being developed under the informal name "universal video coding" (VVC, Versatile Video Coding). The disclosed invention may be used in the context of VVC.

[0051] FIG. 3 may be a functional block diagram of a video decoder (210) according to an embodiment of the present invention.

[0052] The receiver (310) may receive one or more encoded video sequences for decoding by the video decoder (210); in the same or another embodiment, one coded video sequence at a time, where the decoding of each coded video sequence is independent of other coded video sequences. The encoded video sequence may be received from a channel (312), which may be a hardware/software link to a storage device where the encoded video data is stored. The receiver (310) may receive the encoded video data with other data, such as encoded audio data and/or auxiliary data streams, which may be relayed to their respective utilizing entities (not shown). The receiver (310) may separate the encoded video sequence from other data. To combat network jitter, a buffer memory (315) may be connected between the receiver (310) and the entropy decoder/analyzer (320) (hereinafter "analyzer"). When the receiver (310) receives data from a storage/relay device with sufficient bandwidth and controllability, or from an isosynchronous network, buffer memory (315) may not be needed or may be small. For use in best-practice packet networks, such as the Internet, buffer memory (315) may be required, may be relatively large, and may have a predominantly adaptive size.

[0053] The video decoder (210) may include an analyzer (320) for reconstructing symbols (321) from an entropy coded video sequence. The categories of these symbols include information used to control the operation of the video decoder (210) and possibly information to control a visualization device such as a display screen (212) that is not an integral part of the decoder but can be connected to it, as shown in fig. 2. The control information for the rendering device(s) may be Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The analyzer (320) may analyze/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be in accordance with a video coding technology or standard and may follow various principles well known to professionals, including variable run length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The analyzer (320) can extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOP, Groups of Pictures), images, tiles, slices, macroblocks, coding units (CU, Coding Units), blocks, transformation units (TU, Transform Units), prediction units (PU, Prediction Units ) etc. The entropy decoder/analyzer can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0054] The analyzer (320) may perform an entropy decoding/analysis operation of the video sequence received from the buffer memory (315) to create symbols (321).

[0055] Several different modules may be used for symbol reconstruction (521) depending on the type of encoded video image or parts thereof (eg: inter- and intra-pictures, inter- and intra-block) and other factors. Which modules are used and how can be determined by the subgroup management information extracted from the encoded video sequence by the analyzer (320). The flow of such subgroup management information between the analyzer (320) and multiple modules is not shown hereinafter for simplicity.

[0056] In addition to the previously mentioned functional blocks, video decoder 210 may be fundamentally divided into several functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with one another and can, at least partially, be nested within one another. However, for the purposes of describing the disclosed invention, a fundamental subdivision into the following functional modules is appropriate.

[0057] The first module is the scaling/inverse transform module (351). The scaling/inverse transform module (351) receives the quantized transform coefficient as well as control information including the transform used, block size, quantization factor, quantization scaling matrices, and so on. as character(s) (321) from the analyzer (320). Such a module can output blocks containing sample values that can be entered into the aggregator (355).

[0058] In some cases, the output samples of the scaling/inverse transform block (351) may refer to an intra-coded block; that is, a block that does not use predicted information from previously reconstructed images, but may use predicted information from previously reconstructed portions of the current image. Such predictive information may be provided by the prediction module (352) within the picture. In some cases, the intra-image prediction module (352) generates a block of the same size and shape as the block to be reconstructed using previously reconstructed environment information retrieved from the current (partially reconstructed) image buffer (356). The aggregator (355) in some cases adds, for each sample, the prediction information generated by the intra prediction module (352) to the output sample information provided by the scaler/inverse transform module (351).

[0059] In other cases, the output samples of the scaler / demapper (351) may refer to an intra-coded block, possibly with motion compensation. In such a case, the motion compensation prediction module (353) may access the reference picture memory (357) to retrieve samples used for prediction. After motion compensation has been applied to the extracted samples according to the symbols (321) associated with the block, these samples can be added by an aggregator (355) to the output of the scaler/inverse transform module (in this case referred to as residual samples or residual signal) to generate output information. counts. Reference picture memory addresses from where the motion compensation prediction module retrieves predicted samples may be governed by motion vectors available to the motion compensation prediction module in symbol form (321), which may have, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values retrieved from reference picture memory when accurate sub-sample motion vectors, motion vector prediction mechanisms, etc. are used.

[0060] Various loop filtering techniques can be applied to the output samples of the aggregator (355) in the loop filter module (356). The video compression technologies may include deblocking filter technologies under the control of parameters included in the encoded video bitstream and become available to the loop filter module (356) as symbols (321) from the analyzer (320), but may also be responsive to the metainformation received during decoding of the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as to respond to previously reconstructed and contour filtered sample values.

[0061] The loop filter module (356) may provide a stream of samples that may be fed to the renderer (212) as well as stored in reference picture memory (356) for use in future inter-picture prediction.

[0062] Some encoded pictures, when fully reconstructed, can be used as reference pictures for future prediction. For example, when the encoded picture is fully reconstructed and the encoded picture is identified as a reference picture (eg, by analyzer (320)), the current reference picture (356) may become part of the reference picture memory (357), and fresh current picture memory may be re-allocated before reconstruction of the next encoded image.

[0063] Video decoder 320 may perform decoding operations according to a predetermined video compression technology, such as ITU-T Rec. H.265. The encoded video sequence may be consistent with the syntax given by the used video compression technology or standard, in the sense that the encoded video sequence may adhere to both the syntax of the video compression technology or standard according to a video compression technology document or standard, in particular according to this profile document. It may also be necessary for consistency that the complexity of the encoded video sequence remain within the boundaries set by the video compression technology level or standard. In some cases, the levels limit the maximum image size, the maximum frame rate, the maximum sampling rate for reconstruction (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the levels, in some cases, may be further limited by the specifications of the hypothetical reference decoder (HRD, Hypothetical Reference Decoder) and metadata for managing the HRD buffer signaled in the encoded video sequence.

[0064] According to an embodiment, the receiver (310) may receive additional (redundant) data with encoded video. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (320) to correctly decode the data and/or more accurately reconstruct the original video data. The additional data may be, for example, enhancement levels in time, space, or signal-to-noise ratio (SNR), redundant slices, redundant images, forward error correction codes, and so on.

[0065] FIG. 4 may be a functional block diagram of a video encoder (203) according to an embodiment of the present invention.

[0066] The video encoder (203) may receive video samples from a video source (201) (which is not part of the video encoder) that may capture the video image(s) to be encoded by the video encoder (203).

[0067] The video source (201) may provide the source video sequence to be encoded by the video encoder (203) in the form of a digital video sample stream of any suitable bit depth (eg: 8 bit, 10 bit, 12 bit, ...), any color space (eg, BT.601 Y CrCB, RGB, ...) and any suitable sampling structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (201) may be a storage device where previously prepared video is stored. In a videoconferencing system, the video source (203) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual images that give a sense of movement when viewed in sequence. The images themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc. used. It is not difficult for a person skilled in the art to understand the relationship between pixels and samples. The following description is about readings.

[0068] According to an embodiment, the video encoder (203) may encode and compress images of the original video sequence into the encoded video sequence (443) in real time or subject to any other time constraints imposed by the application. Setting the proper coding rate is one of the functions of the controller (450). The controller (450) controls other functional modules as described below and is operatively connected to these modules. The connection is not shown for simplicity. Parameters set by the controller may include parameters related to frequency adjustment (picture skip, quantizer, lambda value applied in rate-distortion optimization, ...), image size, group of pictures scheme (GOP, group of pictures), maximum search area motion vectors, etc. Those skilled in the art are well aware of other functions of the controller (450) that may relate to a video encoder (203) optimized for a particular system design.

[0069] A number of video encoders are configured to operate with a coding loop well known to those skilled in the art. As a very simplified description, an encoding loop may include an encoder coding element (430) ("source encoder") (responsible for creating symbols based on the input image to be encoded and the reference image(s)) and (local ) a decoder (433) built into the video encoder (203) that reconstructs the symbols to create sample data as a (remote) decoder would (because any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies discussed in the disclosed invention). The reconstructed sample stream enters the reference picture memory (434). Since decoding a stream of symbols results in bit-accurate results independent of the position of the decoder (local or remote), the contents of the reference picture memory will also be the same bit-accurate for the local encoder and the remote encoder. In other words, the predicted portion of the encoder "sees" as reference picture samples exactly the same sample values as the decoder would "see" if using prediction during decoding. This fundamental principle of reference picture synchronism (and eventually drift if synchronism cannot be maintained due to, for example, channel errors) is well known to those skilled in the art.

[0070] The "local" decoder (433) may operate in the same manner as the "remote" decoder (210) detailed above with reference to FIG. 3. However, again, referring to FIG. 3, since the symbols are available and the entropy encoder (445) and parser (320) can be losslessly encoded/decoded into the encoded video sequence, the entropy decoding parts of the video decoder (210) including channel (312), receiver (310), buffer memory (315) and parser (320) cannot be fully implemented in the local decoder (433).

[0071] In doing so, it can be concluded that any decoding technology present in a decoder, with the exception of analysis/entropy decoding, must also be present, in essentially identical functional form, in the corresponding encoder. For this reason, the disclosed invention is focused on the operation of the decoder. The description of encoding technologies can be abbreviated as they are the inverse of the decoded technologies described in detail. Only in some areas a more detailed description is required, which is given below.

[0072] In operation, source encoder (430) may perform motion-compensated predictive coding, in which an input picture is predictively encoded based on one or more previously encoded pictures from the video sequence, referred to as "reference pictures". Thus, the encoding engine (432) encodes the differences between the pixel blocks of the input image and the pixel blocks of the reference image(s) that can be selected as the predicted input image reference(s).

[0073] The local video decoder (433) may decode the encoded video data of pictures that can be specified as reference pictures based on the symbols generated by the source encoder (430). The operations of the encoding engine (432) may be predominantly lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (433) duplicates the decoding processes that may be performed by the video decoder on the reference pictures and may cause the reconstructed reference pictures to be stored in the reference picture cache (434). Thus, the video encoder (403) can locally store copies of the reconstructed reference pictures having the same content as the reconstructed reference pictures to be received by the video decoder at the receiver side (in the absence of transmission errors).

[0074] The predictor (435) may perform prediction searches for the coding engine (432). Thus, for a new image to be encoded, the predictor (435) can search the reference image memory (634) for sample data (as candidates for the role of reference pixel blocks) or some metadata, for example, motion vectors of the reference image, block shapes etc., which can serve as a proper reference for predicting new images. The predictor (435) may operate on a sample-block-pixel-block basis in order to find appropriate references for prediction. In some cases, according to the search results obtained by the predictor (435), the input image may have prediction references extracted from a plurality of reference pictures stored in the reference picture memory (434).

[0075] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[0076] The output of all of the above functional modules may be entropy encoded in an entropy encoder (445). The entropy encoder translates symbols generated by various functional modules into an encoded video sequence by lossless symbol compression according to techniques known to those skilled in the art, such as Huffman coding, variable run length coding, arithmetic coding, and so on.

[0077] The transmitter (440) may buffer the encoded video sequence(s) created by the entropy encoder (445) in preparation for transmission over a communication channel (460), which may be a hardware/software link to a storage device where encoded video data. The transmitter (440) may combine the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (sources not shown).

[0078] The controller (450) may control the operation of the video encoder (203). During encoding, the controller (450) may assign to each encoded picture one or another encoded picture type, which may determine the encoding methods applicable to the corresponding picture. For example, images can often be assigned the following image types:

[0079] An intra-picture (1-picture) that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, pictures in the Independent Decoder Refresh ("IDR" format). One skilled in the art will be aware of the 1-image varieties and their respective uses and features.

[0080] A predicted picture (P-picture) that can be encoded and decoded using intra prediction or inter prediction using at most one motion vector and a reference index to predict sample values of each block.

[0081] A bi-directionally predicted picture (B-picture) that can be encoded and decoded using intra prediction or inter prediction using no more than two motion vectors and reference indices to predict sample values of each block. Likewise, multi-predicted pictures may use more than two reference pictures and associated metadata to reconstruct a single block.

[0082] Source images typically allow for spatial division into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coding on a block-by-block basis. The blocks may be predictively encoded with reference to other (previously encoded) blocks determined by the coding assignment applied to the respective images of those blocks. For example, blocks of I-pictures may be encoded without prediction or predictively with reference to previously coded blocks of the same picture (spatial prediction or intra prediction). P-picture pixel blocks may be encoded without prediction, either by spatial prediction or temporal prediction with reference to one previously encoded reference picture. Blocks of B-pictures may be encoded without prediction, by spatial prediction or temporal prediction with reference to one or two previously encoded reference pictures.

[0083] The video encoder (203) may perform encoding operations according to a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (203) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Therefore, the encoded video data may conform to the syntax given by the video coding technology or standard being used.

[0084] According to an embodiment, the transmitter (440) may transmit additional data with encoded video. Video encoder (430) may include such data as part of an encoded video sequence. The additional data may include temporal/spatial/SNR enhancement levels, other forms of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, and so on.

[0085] Before a detailed description of a number of aspects of the disclosed invention, it is necessary to introduce a few terms that will be referred to in the remainder of this document.

[0086] In some cases, a sub-image is a rectangular arrangement of samples, blocks, macroblocks, coding units, or similar semantically combined entities that can be encoded on their own in a different resolution. One or more sub-images may comprise an image. One or more encoded sub-images may constitute an encoded image. One or more encoded sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In some environments, one or more coded sub-images may be assembled into a compressed region without recoding at the sample level in the coded picture, and in similar or other cases, one or more coded sub-images may be extracted from the coded picture in the compressed region.

[0087] Further, adaptive resolution change (ARC, Adaptive Resolution Change) means mechanisms for changing the resolution of an image or sub-image in an encoded video sequence, for example, by resampling a reference image. In the following, the ARC parameters denote control information which is necessary for adaptive resolution change and which may include, for example, filter parameters, scaling factors, output and/or reference picture resolutions, various control flags, and so on.

[0088] The above description focuses on encoding and decoding a single semantically independent encoded video image. Before describing the essence of encoding/decoding multiple sub-pictures with independent ARC parameters and hidden additional complexity, ARC parameter signaling options will be presented.

[0089] FIG. 5A-E show several new options for ARC parameter signaling. Each option has certain advantages and disadvantages in terms of coding efficiency, complexity, and architecture. For ARC parameter signaling, a video coding standard or technology may choose one or more of these options, or options from the prior art. The options must be compatible, and in principle they can be interchanged based on the needs of the application, the standards of the applied technology, or the choice of encoder.

[0090] ARC parameter classes may include:

- resolution increase/decrease in X and Y coordinates individually or in combination,

- resolution up/down ratios with the addition of a temporal dimension indicating an increase/decrease in a series of images at a constant rate,

[0091] - any of the above groups of coefficients may include encoding one supposedly short syntactic element or several such elements, which may indicate a table containing the coefficient (s)

- resolutions in x and y coordinates, sample units, blocks, macroblocks, coding units, or any other suitable element of the input/output picture, reference picture, encoded picture (if there are multiple resolutions (for example, one resolution for the input picture, another for the reference image) in some cases, one set of values can be obtained from another set of values. This can be controlled, for example, using flags. The following is a more detailed example),

- "warping" coordinates, like those used in H.263 Annex P, again in suitable detail as presented above (H.263 Annex P defines one efficient way to encode such coordinates, but other potentially more efficient methods For example, according to embodiments, the reversible variable length, Huffman code encoding of Annex P warp coordinates is replaced by binary encoding of a suitable length, where the length of the binary codeword can be obtained, for example, from the maximum image size, possibly multiplied by a certain factor and offset by a certain value so that the deformation occurs beyond the maximum image size) and/or

- parameters of filters for increasing/decreasing resolution. In the simplest case, one filter is applied to increase / decrease the resolution. However, in some cases it will be more efficient to give flexibility to the filter, which may require filter parameter signaling. Such parameters can be selected through the index of the list of possible filter designs. The filter may be fully specified (eg, via a list of filter coefficients and suitable entropy coding techniques), the filter may be implicitly selected via upscaling/downscaling factors signaled by any of the above mechanisms, and so on.

[0092] The following description assumes encoding a fixed number of upscaling/downscaling coefficients (the same coefficient can be applied in X and Y coordinates) denoted by a codeword. This codeword may advantageously be variable length encoded, eg, by means of an Ext-Golomb code specific to certain syntax elements in the video coding characteristics, eg, for H.264 and H.265. A suitable scaling of the values of the upscaling/downscaling factors can be selected, for example, according to the following table

[0093]

[0094] A plurality of such displays can be modeled according to the needs of the application and the capabilities of resolution up/down mechanisms in a video compression technology or standard. Other values may be presented in the table. Values may also be represented by entropy encoding mechanisms that do not include Ext-Golomb codes, such as binary encoding. Such mechanisms may have certain advantages when considering resampling factors outside of video processing engines (mainly encoder and decoder), for example, via MANE. It should be noted that for the (presumably) most typical case where a resolution change is not required, the Ext-Golomb short code can be chosen. In the table above, this code is represented by one bit. For the most typical case, such a code is more efficient than binary codes.

[0095] The number of entries in the table, as well as their semantics, can be configured in whole or in part. For example, the table base can be specified by a high-level parameter set, such as a sequence or decoder parameter set. Alternatively, or additionally, one or more such tables may be defined in a video coding technology or standard. The tables may be selected by a set of decoder parameters or a sequence.

[0096] Next, we describe a method for incorporating an upscaling/downscaling factor (ARC data) encoded by the above technique into a video coding technology or standard syntax. Similar techniques can be applied to one or more codewords that control up/down filters. The following is a description for a situation where relatively large amounts of data are needed for a filter or other data structures.

[0097] According to the example of FIG. 5A, the image (500A) shows the H.263 Annex P ARC data 502 incorporation as four image header warp coordinates 501, more specifically in the H.263 PLUSPTYPE header extension (503). The choice of such a design is justified by a) the presence of an image header and b) the expectation of frequent changes in the ARC data. However, overhead can be quite large with H.263 signaling, and scaling factors can be out of picture bounds because the picture header may be temporary. Further, according to the example of image 5B, image (500B) shows the inclusion of PPS data (504), ARC reference data (505), SPS data (507), as well as final resolution table data (506) of the JVET-M0135 standard.

[0098] According to exemplary embodiments, FIG. 5C shows an example (500C) with tile group header data (508) and ARC data (509); fig. 5D shows an example (500D) with tile group header data (514), ARC reference data (513), SPS data (516), and ARC data (515), and FIG. 5E shows an example (500E) with adaptation parameter set(s) (APS) data (511) and ARC data (512).

[0099] JVCET-M135-v1 includes ARC reference data (505) (pointer) located in the image parameter set (504), table indexing (506) with finite resolutions located in the sequence parameter set (507). Placing a candidate permission in a table (506), in a sequence parameter set (507), can be justified, according to the oral statements of the authors, when using the SPS as a compatible element in the exchange of capabilities. The resolution may vary within the limits given by the values of the table (506) from image to image when referring to the appropriate set of image parameters (504).

[0100] Referring again to FIG. 5, the following additional options are possible for transmitting ARC data in a video bitstream. Each of these options has certain advantages over the technology described above. Options may be present in the same video coding standard or technology at the same time.

[0101] In an embodiment, the ARC data (509), for example, the resampling (zoom) factor, may be present in a slice header, a GOB header, a tile header, or a tile group header (hereinafter tile group header) (508). This is appropriate when the amount of ARC data is small, such as a fixed length ue(v) variable or a fixed length codeword of a few bits as shown in the above example. The presence of ARC data in the tile group header provides the additional benefit of applying the ARC data to a sub-image represented by, for example, the above tile group rather than the whole image. See also below. In addition, if a video compression technology or standard provides for changes in the adaptive resolution of only the whole image (as opposed to, for example, changes in the adaptive resolution of a group of tiles), inserting ARC data into the tile group header, as opposed to inserting the above data into the H. 263 has certain advantages in terms of error tolerance.

[0102] In the same or another embodiment, the ARC data (512) may be present in the appropriate parameter set (511), for example, in the image parameter set, title, tile, in the adaptation parameter set, etc. (A set of adaptation parameters is presented). The scale of such a set of parameters should preferably not exceed the image, for example, a group of tiles. The use of ARC data is carried out indirectly, through the activation of the desired set of parameters. For example, if a video coding technology or standard considers ARC only within the scope of an image, then a set of image parameters or an equivalent set is appropriate.

[0103] In the same or another embodiment, the ARC reference data (513) may be present in a tile group header (514) or in a similar data structure. Such reference information (513) may be represented by a subset of ARC data (515) available in a parameter set (516) on a scale larger than a single image, such as in a sequence parameter set or in a decoder parameter set.

[0104] An additional layer of indirect and implicit activation of PPS from the tile group header, PPS, SPS used in the JVET-M0135-v1 standard seems optional, according to exemplary embodiments, since image parameter sets, like sequence parameter sets, can be applied ( and are used in some standards, such as RFC3984) to negotiate capabilities or declarations. But if the ARC data is to be applied to a sub-image represented, for example, by tile groups, by a tile group-limited activation parameter set, then an adaptation parameter set or a header parameter set may be an optimal choice. Also, if the size of the ARC data tends to zero (for example, contains filter data - a series of filter coefficients), then the parameter may be a better choice than the header (508) in terms of encoding efficiency, since these settings can be reapplied by other images or sub-images by referring to the same set of parameters, according to exemplary embodiments.

[0105] When using a sequence parameter set or other higher parameter set with multiple images of a certain scale, certain methods can be applied:

1. A set of parameters for storing the data table (516) ARC in some cases can be represented by a set of sequence parameters, and in other cases mainly by a set of decoder parameters. A set of decoder parameters can activate multiple coded video streams (CVS), ie all coded video bits from session start to session end. Such an activation may be more appropriate, since the possible ARC coefficients may be a decoder function, possibly implemented in a hardware device, and hardware functions are usually not changed by any CVS (which, at least in some multimedia systems, are groups of pictures up to one seconds). In this case, embedding a table in a sequence parameter set is explicitly included in the placement options described here, especially in connection with paragraph 2 below.

2. The ARC reference data (513) may advantageously be placed directly in the image/tile slice/GOB/tile group header (hereinafter referred to as the tile group header) (514) rather than in the picture parameter set as in JVCET-M0135-v1. The reason is that when an encoder wants to change one value in a set of image parameters, such as ARC reference data, he has to create a new PPS and links to it. Assume that only the ARC reference data is changed so that, for example, the quantization matrix data remains in the PPS. Such data can be quite large and require redirection to complete the formation of a new PPS. Since ARC reference data (eg, table index (513)) may be represented by a single codeword, which will be the only variable value, it may be cumbersome and useless to retransmit, for example, all data over a quantization matrix. In such a situation, in terms of encoding efficiency, it would be much better to avoid PPS indirection as in JVET-M0135-v1. Similarly, embedding ARC reference data in PPS has an additional disadvantage: ARC reference data (513) is required on the entire image, not on a sub-image, since the image defines boundaries for activating a set of image parameters.

[0106] In the same or other embodiments, ARC parameter signaling may follow the example detailed in FIG. 6. FIG. 6 depicts syntax schemes (600) used in video coding standards. The designation of such syntactic schemes is carried out approximately in the framework of C programming. Lines in bold indicate syntax elements in the bitstream, normal lines often indicate control flow or setting variables.

[0107] A tile group header (601) in the form of an exemplary header syntax structure applied to a (possibly rectangular) sub-image may conditionally include a variable length, the Exp-Golomb (602) encoded dec_pic_size_idx syntax element (depicted in bold). The presence of such a syntax element in the header of a group of tiles can be controlled using adaptive resolution (603) - here the flag value is not displayed in bold. This means that the flag is at a point in the bitstream according to the syntax scheme. It is possible to signal in any high-level syntax structure, inside or outside of the bitstream, whether adaptive resolution is applied to this image or parts of it. In the example shown, the signaling is carried out in a set of sequence parameters as shown below.

[0108] In FIG. 6 shows a fragment of the sequence parameter set (610). The first syntax element introduced is the adaptive image resolution change flag (611). When changed, this flag may indicate the application of an adaptive resolution, which in turn may require certain control data. In the example, such control data is conditionally present based on the value of the flag and the if() statement in the parameter set (612) and the tile group header (601).

[0109] When using adaptive resolution, according to exemplary embodiments, the output resolution is encoded in units of samples (613). The number 613 refers to the output_pic_width and output_pic_height parameters in the brightness samples, which together can determine the resolution of the output image. In other situations, any value can be subject to certain restrictions in the video coding technology or standard. For example, the level definition can limit the number of output samples that can result from the value of those two syntax elements. Also, certain video coding technologies or standards, or external technologies or standards, such as system standards, may restrict the quantitative range (for example, one or more measurements must be divisible by a power of two) or aspect ratio (for example, width and height must be in a ratio of 4: 3 or 16:9). Such restrictions may be introduced to reduce the load on hardware devices or for other reasons.

[0110] In some applications, it may be recommended that the decoder apply a specific reference picture size at the command of the encoder, rather than accepting the default output picture size. In this example, the reference_pic_size syntax element flag (614) controls the relative sizes of the reference pictures (615) (again, the number refers to the width and height). [0111] Finally, a table of possible values for the width and height of the decoded image is shown. Such a table may be represented, for example, by a table index (num_dec_pic_size_in_luma_samples_minusl) (616). The value "minus 1" may denote an interpretation of the meaning of that syntax element. For example, if the encoded value is zero, then there will be one table entry available. If the value is five, then the table will have six entries. Next, for each "row" of the table, the width and height of the decoded image is entered into the syntax (617).

[0112] The rendered table entries (617) may be indexed by the dec_pic_size_idx (602) syntax element in the tile group header, thereby specifying different decoding sizes (essentially scaling factors) for each tile group.

[0113] Certain video coding technologies or standards, such as VP9, support spatial scaling through certain forms of reference picture resampling (signaled differently by the disclosed invention) in conjunction with temporal scaling to enable spatial scaling. In particular, certain reference images can be upsampled using ARC techniques to form a spatial enhancement layer base. For detail, these higher resolution images can be enhanced by standard high resolution prediction mechanisms.

[0114] The disclosed invention can be used in such an environment. In some cases, in the same and other embodiments, a NAL unit header value, such as a temporal ID field, can be used to indicate not only the temporal but also the spatial level. This approach has certain advantages for certain system designs: for example, with existing selective addressing units (SFUs) designed and optimized for selective addressing of the temporal layer based on the NAL unit header, the temporal ID header can be used unchanged for scalable environments. This requires scaling between the encoded picture size and the temporal level indicated by the temporal ID field in the NAL unit header.

[0115] In some video coding technologies, an access unit (AU, Access Unit) may mean encoded image(s), slice(s), tile(s), NAL unit(s), etc. received and composed to the corresponding bitstream of the image/slice/tile/NAL unit of a certain point in time. This moment may be composition time.

[0116] In HEVC and certain video coding technologies, a picture order count (POC) value can be used to indicate a selected reference picture among a plurality of reference pictures stored in a decoded picture buffer (DPB). If an access unit (AU) includes one or more images, slices or tiles, then each image, slice or tile belonging to the same access unit may have the same POC value, from which it can be understood that they were created from the content of the same and the same composition time. In other words, if two images/slices/tiles have an identical setpoint POC, this may mean that the two images/slices/tiles belong to the same access block and have the same composition time. Conversely, two images/slices/tiles with different POC values may indicate that these images/slices/tiles belong to different access blocks and different composition times.

[0117] According to illustrative embodiments of the disclosed invention, such a strong association can be weakened in the sense that the access block can include images, slices or tiles with different POC values. By allowing different POC values within an access block, it becomes possible to use the POC value to identify potentially independent decodable pictures/slices/tiles with the same presentation time. This, in turn, allows multiple scalable levels to be supported without changes to the reference picture selection signaling (eg, reference picture set or list signaling), as described in more detail below.

[0118] However, it is still desirable to be able to identify the AU to which an image/slice/tile belongs relative to other images/slices/tiles with different POC values solely by the POC value. This can be achieved in the following way.

[0119] In the same and other embodiments, an access unit number (AUC) may be signaled in a high-level syntax structure, for example, in a NAL unit header, slice header, group header, SEI message, parameter set, or AU delimiter. The AUC value may be used to identify NAL units, pictures, slices, or tiles belonging to a given access unit. The AUC value may correspond to a certain point in time of the composition. The AUC value can be a multiple of the ROS value. The AUC value can be calculated by dividing the ROC value by an integer. In some cases, division operations can load the decoder. In such cases, slight restrictions on the numbering range of AUC values can be allowed to replace the division operation with shift operations. For example, the AUC value may be equal to the most significant bit (MSB) value of the POC value range.

[0120] In the same and other embodiments, a picture sequence number (ROS) cycle value per access unit (poc_cycle_au) may be signaled by a high-level syntactic structure, e.g., NAL unit header, slice header, tile group header, SEI message, set parameters or limiter AU. The poc_cycle_au parameter may indicate how many different and consecutive POC values can be associated with the same AU. For example, if the value of ROS_cycle_au is four, then images, slices, or tiles with ROS values from 0 to 3 inclusive are associated with AU at an AUC value of zero, and images, slices, or tiles with ROS values from 4 to 7, inclusive, are associated with AU at a value of AUC equal to one. Thus, the AUC value can be obtained by dividing the ROS value by the ROS_cycle_au value.

[0121] In the same and other embodiments, the value of po_cycle_au can be derived from data located, for example, in a Video Parameter Set (VPS) that defines the number of spatial and SNR levels in the encoded video sequence. Such a possible relationship is briefly presented below. Since obtaining in the above manner can save a few bits in the VPS and thereby improve coding efficiency, the same method may be advantageous to explicitly encode the poc cycle au variable in an appropriate high-level syntax structure that is hierarchically below the video parameter set, in order to minimize the poc_cycle_au variable to a given small portion of a bitstream, such as an image. Such an optimization can save more bits compared to the derivation process presented above, since POC values (and/or values of syntax elements not directly related to POC) can be decoded into lower level syntax structures.

[0122] In the same and another embodiment of FIG. 9 shows an example (900) of syntax tables for signaling a vps_poc_cycle_au variable syntax element to a VPS (or SPS) that indicates the application of poc cycle au for all pictures/slices in a coded video sequence, as well as a slice_poc_cycle_au variable syntax element that points to a poc cycle variable. au of the current slice in the slice header. When the value of POC increases uniformly per access block, the value of the variable vps_contant_poc_cycle_per_au in the VPS is set to one, and the variable vps_poc_cycle_au is signaled in the VPS. In this case, slice_poc_cycle_au is not explicitly signaled, and the AUC value for each AU is calculated by dividing the POC value by vps_poc_cycle_au. If the POC value does not increase evenly per access block, the vps_contant_poc_cycle_per_au variable in the VPS is set to zero. In this case, vps_access_unit_cnt is not signaled, but slice_poc_acess_unit_cnt is signaled in the slice header for each slice or picture. Each slice or image can have different slice access unit cnt values. The AUC value for each AU is calculated by dividing the POC value by by slice_poc_cycle_au. Fig. 10 shows a flowchart illustrating a meaningful processing flow (1000) in which S100 - VPS/SPS analysis and determining whether the POC cycle per access unit is a constant value, in step S101, the POC cycle constant per access unit in the coded video sequence is determined. If not, in step S103, an access unit number value is calculated based on the picture level po_cycle_au value and the POC value. If the result is positive, in step S102, an access block number value is calculated from the sequence level poc cycle au value and the POC value. In step S104, the VPS/SPS is analyzed again, and the per-access block POC cycle persistence is determined, which may continue cyclically or otherwise for one or more portions of the processing data stream (1000).

[0123] In the same and other embodiments, even if the POC value of an image, slice, or tile may be different, an image, slice, or tile corresponding to an AU with the same AUC value may be associated with the same decoding or output time. Thus, without parsing/decoding dependency between images, slices or tiles in the same access block, all or a subset of the images, slices or tiles associated with the same access block can be decoded in parallel and output at the same time. .

[0124] In the same and other embodiments, even though the POC value of an image, slice, or tile may be different, an image, slice, or tile corresponding to an AU with the same AUC value may be associated with the same composition/display moment. When the composition time is contained in the container format, even if the images correspond to different AUs, if the images have the same composition time, the images can be displayed at the same point in time.

[0125] In the same and other embodiments, each image, slice, or tile may have the same temporal identifier (temporal_id) in the same access block. All or a subset of images, slices or tiles corresponding to a point in time may be associated with the same temporal sublayer. In the same and other embodiments, each image, slice or tile may have the same or different spatial layer identifier (layer_id) in the same access block. All or a subset of the images, slices or tiles corresponding to a point in time may be associated with the same or a different spatial level.

[0126] FIG. 8 shows an example (800) of a video sequence structure with values of temporal_id, layer_id, POC, and AUC at adaptive resolution change. In this example, the image, slice or tile in the first access block with AUC=0 has temporal_id=0 and layer_id=0 or 1, and the image, slice or tile in the second access block with AUC=1 has temporal_id=1 and layer_id= 0 or 1, respectively. The POC value is incremented by 1 per image, regardless of the temporal_id and layer_id values. In this example, the value of ros_cycle_au can be two. It is preferable to set the value of ros_cycle_au to the number of levels (spatial scalability). Thus, in this example, the ROS value is increased by 2 while the AUC value is increased by 1.

[0127] In exemplary embodiments, all or a subset of inter-picture or inter-layer prediction structures and reference picture indication may be supported by a set of existing reference pictures (RPS? reference picture set) signaled in HEVC or in a reference picture list (RPL) . In RPS or RPL, the selected reference picture is indicated by signaling a POC value or a POC value delta between the current picture and the selected reference picture. For the disclosed invention, RPS and RPL can be used to indicate an inter-picture or inter-layer prediction structure without signaling change, but with the following limitations. If the current picture's temporal_id value is greater than the current picture's layer_id value, then the current picture cannot use the reference picture for motion compensation or other predictions.

[0128] In the same and other embodiments, motion vector scaling based on POC differences for temporal motion vector prediction may be disabled for multiple pictures in an access block. Thus, although each picture may have a different POC value within an access block, the motion vector is not scaled and is used to predict the temporal motion vector within an access block. This is because a reference picture with different ROCs in the same access block is considered as a reference picture with the same point in time. Thus, in exemplary embodiments, the motion vector scaling function may again assume a value of 1 if the reference picture belongs to an access unit associated with the current picture.

[0129] In the same and other embodiments, motion vector scaling based on the difference in POC for temporal motion vector prediction can be optionally disabled in multiple pictures when the spatial resolution of the reference picture differs from the spatial resolution of the current picture. When motion vector scaling is enabled, the above motion vector is scaled based on both the difference in DOC and the spatial resolution ratio between the current and reference image.

[0130] In the same or another embodiment, the motion vector can be scaled based on the difference in AUC instead of the difference in POC to predict the temporal motion vector, especially in the absence of a fixed value for poc_cycle_au (when vps_contant_poc_cycle_per_au=0). In other cases (when vps_contant_poc_cycle_per_au=1), motion vector scaling based on difference in AUC may correspond to motion vector scaling based on difference in POC.

[0131] In the same or different embodiment, when the motion vector is scaled based on the difference in AUC, the reference motion vector in the same access block (with the same AUC value) with the current picture is not scaled based on the difference in AUC and is used for prediction motion vector without scaling or with scaling based on the ratio of spatial resolution between the current and reference images.

[0132] In the same and other embodiments, the AUC value is used to identify the access block boundary and to operate a hypothetical reference decoder (HRD) requiring input and output synchronization with AU granularity. In most cases, the decoded picture with the highest level in the AU can be displayed. The value of AUC and layer_id can be used to identify the output image.

[0133] In exemplary embodiments, an image may consist of one or more sub-images. Each sub-image may represent a portion of an area or an entire area of an image. An area of a sub-image may or may not overlap with an area of another sub-image. An area of one or more sub-images may or may not cover the entire image area. If the image consists of a sub-image, then the sub-image area is identical to the image area.

[0134] In the same and other embodiments, the implementation of the sub-image can be encoded in an encoding method similar to the encoding method of the encoded image. A sub-image may be encoded separately or based on another sub-image or encoded image. A sub-image may or may not depend on the results of analysis of another sub-image or encoded image.

[0135] In the same and other embodiments, the encoded sub-image may be contained in one or more layers. The encoded sub-image may have different spatial resolutions. The original sub-image may be spatially resampled (upsampled or downsampled), encoded with different spatial resolution parameters, and may be in a bitstream that corresponds to a layer.

[0136] In the same and other embodiments, a sub-image with (W, H), where W is the width of the sub-image, H is the height of the sub-image, respectively, can be encoded and included in the encoded bitstream corresponding to level 0, and the sub-image with increased (downgraded) ) resolution relative to the sub-image of the original spatial resolution with W*S _w,k , H*S _h,k ) can be encoded and included in the encoded bitstream corresponding to layer k, where S _w , _k , S _h,k denote the horizontal resampling factor and verticals. If the values of S _w,k , S _h,k are greater than 1, then oversampling means upsampling. If the values of S _w,k , S _h,k are less than 1, then oversampling means reducing the resolution.

[0137] In the same and other embodiments, an encoded sub-image of a layer may have a different image quality compared to an encoded sub-image in a different layer of the same sub-image or a different sub-image. For example, sub-image i in layer n is encoded with quantization parameter Q _i,n , and sub-image j in layer m is encoded with quantization parameter Q _j,m .

[0138] In the same and other embodiments, an encoded sub-image in a layer can be decoded autonomously without parsing or decoding an encoded sub-image in another layer of the same area. A sub-image layer that can be decoded autonomously without referring to another sub-image layer of the same area is an autonomous sub-image layer. An encoded sub-image in an offline sub-image layer may or may not depend on decoding or parsing a previous encoded sub-image in the same sub-image layer, but a coded sub-image is independent of a coded image in another sub-image layer.

[0139] In the same and other embodiments, an encoded sub-image in a layer may be decoded based on parsing or decoding an encoded sub-image in another layer of the same local area. A sub-image layer that can be decoded with reference to another sub-image layer of the same local area is a dependent sub-image layer. An encoded sub-image in a dependent sub-image may refer to an encoded sub-image belonging to the same sub-image, a previous encoded sub-image in the same sub-image layer, or to both reference sub-images.

[0140] In the same and other embodiments, an encoded sub-image consists of one or more independent sub-image layers and one or more dependent sub-image layers. However, an encoded sub-image may have at least one independent sub-image layer. An independent sub-image layer may have a layer identifier (layer_id) value, which may be present in a NAL unit header or other high-level syntax structure, equal to zero. The subimage layer with layer_id equal to zero is the base subimage layer.

[0141] In the same and other embodiments, an image may be composed of one or more foreground sub-images and one background sub-image. The area supported by the foreground sub-image may overlap with the area supported by the background sub-image. The background sub-image may be the base level of the sub-image, and the foreground sub-image may be the non-base level (enhancement level) of the sub-image. One or more non-base sub-image layers may refer to the same base layer for decoding. Each non-base subimage layer with layer_id equal to a may refer to a non-base subimage layer with layer_id equal to b, where a is greater than b.

[0142] In the same and other embodiments, an image may be composed of foreground sub-images with or without a background sub-image. Each sub-image may have its own base sub-image level and one or more non-base levels (enhancement levels). Each base sub-image level may be a reference to one or more non-base sub-image levels. Each non-base subimage layer with layer_id equal to a may refer to a non-base subimage layer with layer_id equal to b, where a is greater than b.

[0143] In the same and other embodiments, an image may be composed of one or more foreground sub-images with or without a background sub-image. Each encoded sub-image in a (base or non-base) sub-image layer may be a reference for one or more non-basic layer sub-images belonging to the same sub-image and one or more non-basic layer sub-images that do not belong to the same sub-image.

[0144] In the same and other embodiments, an image may be composed of one or more foreground sub-images with or without a background sub-image. The sub-image in layer a can be divided into a plurality of sub-images in the same layer. One or more encoded sub-images in layer b may refer to a split sub-image in layer a .

[0145] In the same and other embodiments, a coded video sequence (CVS) may be a group of coded pictures. The CVS may consist of one or more coded sub-picture sequences (CSPS), where the CSPS may be a group of coded sub-pictures covering the same local area of an image. The CSPS may have the same or different temporal resolutions compared to the encoded video sequence.

[0146] In the same and other embodiments, the CSPS may be encoded and be in one or more layers. The CSPS may consist of one or more levels of CSPS. Decoding one or more CSPS levels corresponding to the CSPS may reconstruct a sequence of sub-images corresponding to the same local area.

[0147] In the same and other embodiments, the number of CSPS levels corresponding to a CSPS may be identical or different from the number of CSPS levels corresponding to another CSPS.

[0148] In the same or another embodiment, the CSPS layer may have a temporal resolution (eg, frame rate) that is different from the temporal resolution of another CSPS layer. The original (uncompressed) sub-image sequence may be temporally resampled (upsampled or downsampled), encoded with different temporal resolution parameters, and be in a bitstream that matches the level.

[0149] In the same or another embodiment, a sub-image sequence with a frame rate F may be encoded and contained in an encoded bitstream corresponding to layer 0, and the sub-image sequence with increased (or reduced) resolution of the original sub-image sequence with F*S _t,k may be encoded and contained in the encoded bitstream corresponding to layer k, where S _t,k denotes the sample timing for layer k. If the value of S _t,k is greater than 1, then the temporal resampling process corresponds to frame rate upconversion. If the value of S _t,k is less than 1, then the temporal resampling process corresponds to a frame rate down conversion.

[0150] In the same or other embodiments, when a CSPS layer a sub-image is a reference to a CSPS layer b sub-image for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b, the decoded pixels in CSPS layer a are resampled and used as reference. The resampling process may require filtering to increase or decrease resolution.

[0151] FIG. 11 shows an example video stream (1100) including a background CSPS video with a layer_id value of zero and multiple foreground CSPS layers. Since a coded sub-image may be composed of one or more CSPS layers, a background region that does not belong to any foreground CSPS layer may be composed of a base layer. The base layer may include a foreground and background region, and the CSPS enhancement layer contains the foreground region. The CSPS enhancement layer may refer to the reconstructed pixels and motion vectors of the base layer corresponding to the same area.

[0152] In the same or other embodiments, the video bitstream corresponding to the base layer is in a track, and the CSPS levels corresponding to each sub-image are in a separate track, in the video file.

[0153] In the same and other embodiments, the video bitstream corresponding to the base layer is in a track, and the CSPS layers with the same layer_id variable are in a separate track. In this example, the track corresponding to level k includes CSPS levels corresponding exclusively to level k.

[0154] In the same or other embodiments, each CSPS layer of each sub-image is stored in a separate track. Each track may or may not depend on the analysis or decoding of one or more other tracks.

[0155] In the same or other embodiments, each track may contain bitstreams corresponding to level j of CSPS levels of all or a subset of sub-images, where 0<i=<j=<k, with k being the highest CSPS level.

[0156] In the same and other embodiments, an image consists of one or more sets of associated media data, including a depth map, an alpha map, 3D configuration data, an occupancy map, and so on. Such time-related media data can be divided into one or multiple sub-data streams, each of which corresponds to one sub-image.

[0157] In the same and other embodiments of FIG. 12 shows an example of videoconferencing (1200) based on the layered sub-image method. The video stream contains one base video bitstream level corresponding to the background image and one or more video bitstreams with enhancement levels corresponding to the foreground sub-images. Each video bitstream of the enhancement layer corresponds to a CSPS layer. On the screen, the image corresponding to the baseline is displayed by default. It contains one or more custom pictures in picture (PIP). When a particular user is selected by the client protocol, the CSPS enhancement level corresponding to the selected user is decoded and displayed in enhanced quality or spatial resolution. Fig. 13 shows an operation (1300) in which, in step S130, a multilayer video bitstream is decoded, and in step S131, a background area and one or more foreground sub-images are identified. In step S132, it is determined whether a particular area of the sub-image has been selected. If not, in step S134, the background area is decoded and displayed. If a specific area has been selected, then in step S133, the enhanced sub-image is decoded and displayed, and the circuit (1300) may continue cyclically or be performed sequentially or in parallel with other operations.

[0158] In the same and other embodiments, a central network device (eg, a router) may select a subset of layers to send to the user depending on the bandwidth of the device. The picture/sub-picture configuration can be used for bandwidth adaptation. For example, if the user does not have enough bandwidth, then the router removes layers or selects a number of sub-images based on their importance or based on the settings applied. You can do this dynamically to adapt to the bandwidth.

[0159] FIG. 14 shows an example of a 360° video (1400). When a 360° spherical image is projected onto a planar image, the 360° projection image can be divided into a plurality of sub-images as a base layer. The enhancement level of an individual sub-image may be encoded and transmitted to a client. The decoder may decode the base layer including all sub-pictures and the enhancement layer of the selected sub-picture. When the current viewport is identical to the selected sub-image, the rendered image may be upscaled when the sub-image is decoded with the enhancement level. Otherwise, a low quality decoded image with a base layer is displayed.

[0160] In the same and other embodiments, any layout data for display can be represented in the file as additional information (as an SEI message or metadata). One or more decoded sub-pictures may be moved and displayed based on the signaled layout data. The layout data can be signaled by a streaming server, a broadcaster, or retrieved by a network entity, a cloud server, or defined by user settings.

[0161] In exemplary embodiments, when an input image is divided into one or more (rectangular) subregions, each subregion may be encoded as an independent layer. Each independent layer corresponding to the local area can have a unique value for the layer_id variable. The sub-image size and location data may be signaled for each independent layer. For example, image size (width, height), data about the offset of the upper left corner (x_offset, y_offset). Fig. 15 shows an arrangement example of divided sub-images (1500), sub-images size, position data, and corresponding image prediction structure. Layout data, including sub-image size(s) and sub-image location(s), may be signaled in a high-level syntax structure, for example, in parameter set(s), a slice or tile group header, or in an SEI message.

[0162] In the same and other embodiments, each sub-image corresponding to the independent layer may have a unique POC value in the access block. When the reference picture among the pictures stored in the decoded picture buffer (DPB) is indicated by the syntax element(s) in the RPS or RPL structure, the POC value(s) of each sub-picture that corresponds to the layer can be applied.

[0163] In the same and other embodiments, to indicate the prediction structure (between layers), the layer_id variable cannot be used, but the POC (delta) value can.

[0164] In the same and other embodiments, a sub-image with a POC value of N corresponding to a level (or a local area) may or may not be used as a sub-image reference image with a POC value of N+K corresponding to the same level (or one and the same local area) for motion compensation prediction. In most cases, the value of K may correspond to the maximum number of (independent) levels, which may be equal to the number of subregions.

[0165] In the same and other embodiments of FIG. 16 shows an extended case (1600) for the example of FIG. 15. When dividing the input image into multiple (eg, four) sub-regions, each local region may be encoded with one or more levels. In this case, the number of independent levels may be equal to the number of sub-regions, and one or more levels may correspond to a sub-region. Thus, each sub-region can be encoded with one or more independent levels and zero or more dependent levels.

[0166] In the same and other embodiments (FIG. 16), the input image can be divided into four sub-regions. The top sub-region on the right can be coded as two levels, level 1 and 4, and the right-bottom sub-region can be coded as the following two levels: level 3 and 5. In this case, level 4 may refer to level 1 for motion compensation prediction, and level 5 may refer to to level 3 for motion compensation.

[0167] In the same and other embodiments, loop filtering (e.g., deblocking filtering, adaptive loop filtering, shaping, two-way filtering, or any deep learning based filtering) on the level boundary can be disabled (optionally).

[0168] In the same and other embodiments, motion compensation prediction or intra-block copying at the level boundary can be disabled (optionally).

[0169] In the same and other embodiments, boundary padding can optionally be performed to predict motion compensation or contour filtering at the sub-image boundary. A flag indicating boundary padding (no padding) may be signaled in a high-level syntax structure, such as in a parameter set(s) (VPS, SPS, PPS or APS), a slice or tile group header, or in an SEI message.

[0170] In the same and other embodiments, the layout data of the sub-area(s) (or sub-image(s)) can be signaled in the VPS or SPS. Fig. 17 shows an example (1700) of syntax elements in VPS and SPS. In this example, the variable vps_sub_picture_dividing_flag is signaled to the VPS. The flag may indicate whether the input image is divided into multiple sub-regions or not. When the value of vps_sub_picture_dividing_flag is 0, the input picture(s) in the encoded video sequence corresponding to the current VPS cannot be divided into a plurality of sub areas. In this case, the size of the input image may be equal to the size of the encoded image (pic_width_in_luma_samples, pic_height_in_luma_samples) that is signaled in the SPS. When the value of vps_sub_picture_dividing_flag is 1, the input(s) image(s) may be divided into a plurality of sub-areas. In this case, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled to the VPS. The values vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma samples may be equal to the width and height of the input image(s), respectively.

[0171] In the same and other embodiments, the values of the variables vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples cannot be used for decoding, but can be used for composition and display.

[0172] In the same and other embodiments, when the value of the variable vps_sub_picture_dividing_flag is equal to 1, the syntax elements pic_offset_x and pic_offset_y can be signaled in the SPS, which corresponds to (a) the specific layer(s). In this case, the encoded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to the specific layer. Also, the position (pic_offset_x, pic_offset_y) of the upper left corner of the sub-area may be signaled in the SPS.

[0173] In the same and other embodiments, the location data (pic_offset_x, pic_offset_y) of the upper left corner of the sub-area cannot be used for decoding, but can be used for composition and display.

[0174] In the same and other embodiments, the location data (size and position) of all or a subset of the area(s) of the input(s) image(s), the relationship data between layer(s) may be signaled in a parameter set or SEI message. Fig. 18 shows an example (1800) for specifying sub-area placement data, a relationship between levels, and a relationship between a sub-area and one or more levels. In example (1800), the num_sub_region syntax element indicates the number of (rectangular) sub-regions in the current encoded video sequence. The num_layers syntax element indicates the number of layers in the current encoded video sequence. The num_layers value can be equal to or greater than the num_sub_region value. When any sub-region is encoded as one level, the value of num_layers may be equal to the value of num_sub_region. When one or more sub-regions are encoded as a set of levels, the num_layers index may be greater than the num_sub_region index. The direct_dependency_flag[i][j] syntax element indicates a dependency from level j to level i. The variable num_layers_for_region[i] indicates the number of levels associated with subregion i. The variable sub_region_layer_id[i][j] indicates the relationship of the layer_id of layer j to subregion i. The variables sub_region_offset_x[i] and sub_region_offset_y[i] indicate the horizontal and vertical position of the upper left corner of subregion i, respectively. Sub-region width [i] and sub-region height [i] indicate the width and height of sub-region i, respectively.

[0175] In one embodiment, one or more syntax elements denoting an output layer set for specifying one or more layers to be output with or without profile tier data may be signaled in a high-level syntax structure, e.g., in VPS, DPS, SPS, PPS, APS or SEI message. According to example (1900) of FIG. 19, a num_output_layer_sets syntax element indicating an output layer set (OLS) number in a coded video sequence with reference to a VPS may be signaled to the VPS. For each set of output levels, the variable output_layer_flag can be signaled multiple times, according to the number of output levels.

[0176] In the same and other embodiments, the output_layer_flag[i] variable equal to 1 means that layer i is output, the Variable vps_output_layer_flag[i] equal to zero means that layer i is not output.

[0177] In the same and other embodiments, one or more syntax elements pointing to profile tier data for each set of output levels may be signaled in a high level syntax structure, such as in a VPS, DPS, SPS, PPS, APS, or SEI message. Again, according to FIG. 19, the num_profile_tile_level syntax element indicating the number of profile tier data in each OLS encoded video sequence with reference to the VPS may be signaled to the VPS. For each set of output levels, a set of syntax elements for profile level data or a specific profile level data pointer among profile layer data records may be signaled multiple times according to the number of output levels.

[0178] In the same and other embodiments, the variable profile_tier_level_idx[i][j] denotes a pointer in the profile_tier_level() list of VPS syntax structures, a pointer to the profile_tier_level() syntax structure applied to level j of output level set i.

[0179] In the same and other embodiments, according to Example (2000) of FIG. 20, the syntax elements num_profile_tile_level and/or num output layer sets may be signaled when the number of maximum levels is greater than one (vps_max_layers_minus 1>0).

[0180] In the same and other embodiments, as shown in FIG. 20, the vps_output_layers_mode[i] syntax element indicating the output layer signaling mode for the output layer set i may be present in the VPS.

[0181] In the same and other embodiments, the variable vps_output_layers_mode [i] equal to zero indicates that the output is exclusively the highest layer in the set of output layers i. The variable vps_output_layer_mode[i] equal to one means that all levels are output in the set of output levels i. The variable vps_output_layer_mode[i] equal to 2 means that the output levels are those with the vps_output_layer_flag[i][j] flag equal to 1 in the set of output levels i. Other values are possible, according to embodiments.

[0182] In the same and other embodiments, the output_layer_flag[i][j] variable may or may not be signaled depending on the value of the vps_output_layers_mode[i] variable for output layer set i.

[0183] In the same and other embodiments, as shown in FIG. 20, the flag vps_ptl_signal_flag[i] may be present for the set of output levels i. Depending on the value of vps_ptl_signal_flag[i], the layer profile data for output layer set i may or may not be signalled.

[0184] In the same and other embodiments, with reference to FIG. 21, the number of sub-pictures, max_sub_pics_minusl, in the current CVS may be signaled in a high-level syntax structure, such as in a VPS, DPS, SPS, PPS, APS, or SEI message.

[0185] In the same and other embodiments, as shown in FIG. 21, a sub-picture identifier, sub_pic_id[i] for a sub-picture may be signaled when the number of sub-pictures is greater than one (max_sub_picsminus 1>0).

[0186] In the same and other embodiments, one or more syntax elements indicating a sub-image identifier belonging to each layer of each output layer set may be signaled to the VPS. According to FIG. 21, the sub_pic_id_layer[i][j][k] variable pointing to the sub-image k is present in layer j of the output layer set i. With this data, the decoder can recognize sub-pictures that can be decoded and output for each level of a particular set of output levels.

[0187] In the same and other embodiments, the following syntax elements may be used to define the arrangement of sub-images in layers or in a single layer. Sub-image split output layer sets can be signaled with profile/tier/layer data in a VPS or SPS. Updated sub-picture layout data may be present in the PPS when the picture size is updated by resampling the reference picture. For VPS, you can consider table 2:

[0188] According to exemplary embodiments (Table 2), the variable vps_sub_picture_info_present_flag equal to 1 indicates that syntax elements pointing to the sub-picture layout and identifiers are present in the VPS. The variable vps_sub_picture_info_present_flag equal to zero means that the sub-picture layout syntax elements and identifiers are not present in the VPS.

[0189] According to exemplary embodiments (Table 2), a variable vps sub_pic id_present_flag equal to 1 indicates that vp_ sub_pic_id [i][j] is present in the VPS. The variable vps sub_pic_id_present_flag equal to zero means that vps_sub_pic_id[i][j] is not present in the VPS.

[0190] According to exemplary embodiments (Table 2), the variable vps_sub_pic_id_length_minus 1 plus one denotes the number of bits used to display the vps_sub_pic_id[i][j] syntax element. vps_sub_pic_id_length_minus 1 must be between 0 and 15 inclusive. If this condition is not met, the value of vps_sub_pic_id_length_minusl is assumed to be Ceil(Log2(Max(2, vps_num_sub_pic in_pic_minus 1[i]+1))) 1 for level i.

[0191] According to exemplary embodiments (Table 2), the variable vps_sub_pic_id[i][j] denotes a sub-picture identifier] of level i. The length of the vps_sub_pic_id[i][j] syntax element is vps_sub_pic_id_length_minus 1+1 bits. If this condition is not met, the value of vps_sub_pic_id[i][j] is assumed to be j for each j in the range from 0 to vps_num_sub_pic in_pic minus 1[i] inclusive.

[0192] According to exemplary embodiments (Table 2), the variable vps_pic_width_max_in_luma_samples[i] denotes the maximum width in units of luma samples of each decoded layer i image. The variable pic_width_max_in_luma_samples must not be zero and must be an integer multiple of MinCbSizeY.

[0193] According to the exemplary embodiments (Table 2), the variable pic_height_max_in_luma_samples denotes the maximum height in units of luma samples of each decoded picture referring to the SPS. The variable pic_height_max_in_luma_samples must not be zero and must be an integer multiple of MinCbSizeY.

[0194] According to the exemplary embodiments (Table 2), the variable vps sub_pic_offset_x_in_luma_samples[i][j] denotes the horizontal offset (in units of luma samples) of the upper left luma sample of the sub-image] level i relative to the luminance sample of the upper left corner of the composite image. If this condition is not met, vps_sub_pic_offset_x_in_luma_samples[i][j] is assumed to be zero. The value of vps_sub_pic_offset_x_in_luma_samples[i][j] must be an integer multiple of the coding tree block size.

[0195] According to the exemplary embodiments (Table 2), the variable vps_sub_pic_offset_y_in_luma_samples[i][j] denotes the vertical offset (in units of luminance samples) of the upper left luma sample of the subimage] level i relative to the luminance sample of the upper left corner of the composite image. If this condition is not met, vps_subjtic_offset_y_in_luma_samples[i][j] is assumed to be zero. The value of vps_sub_pic_offset_y_in_luma_samples[i][j] must be an integer multiple of the coding tree block size.

[0196] According to exemplary embodiments (Table 2), the variable vps_sub_pic_width_in_luma_samples[i][j] denotes the sub-image width] of level i in units of luminance samples. The variable vps_sub_pic_width_in_luma_samples[i][j] must be an integer multiple of the coding tree block size.

[0197] According to exemplary embodiments (Table 2), the variable vps_sub_pic_height_in_luma_samples[i][j] denotes the sub-image height] of level i in units of luminance samples. The variable vps_sub_pic_height_in_luma_samples[i][j] must be an integer multiple of the coding tree block size.

[0198] According to exemplary embodiments (Table 2), the variable vps_num_output_layer_sets_minusl plus one denotes the number of output layer sets in the coded video sequence referring to the VPS. If this condition is not met, vps_num_output_layer_sets_minus 1 is assumed to be zero.

[0199] According to exemplary embodiments (Table 2), the variable vps_num_profile_tile_levels_minusl plus one denotes the amount of profile/tile/level data in the encoded video sequence with reference to the VPS. If this condition is not met, vps_num_profile_tile_levels_minusl is assumed to be zero.

[0200] According to exemplary embodiments (Table 2), the variable vps_output_layers_mode [i] equal to zero means that only the highest layer in the output layer set i is output. The variable vps_output_layer_mode[i] equal to one means that all levels are output in the set of output levels i. The variable vps_output_layer_mode[i] equal to two means that the output levels are levels with vps_output_layer_flag[i][j] equal to one in the set of output levels i. vps_output_layers_mode[i] must be between 0 and 2 inclusive. The value 3 of vps_output_layer_mode[i] is retained for future use by ITU-T | ISO/IEC.

[0201] According to exemplary embodiments (Table 2), the variable vps_num_output_subpic_layer_minusl[i][j] means the number of layer subimages] in output layer set i.

[0202] According to exemplary embodiments (Table 2), the variable vps_sub_pic_id_layer[i][j] [k] means the subpicture ID of the output subpicture k subpicture] of layer i. The syntax element length is vps_sub_pic_id_length_minus 1+1 bits. If this condition is not met, vps_sub_pic_id_layer[i][j] [k] is assumed to be k for each j in the range 0 to num_output_subpic_layer_minusl[i][j] inclusive.

[0203] According to exemplary embodiments (Table 2), the variable vps_output_layer_flag[i][j] equal to one means that level] of the set of output levels i is output. The variable vps_output_layer_flag[i] [j] equal to zero means that level j of the set of output levels i is not an output.

[0204] According to exemplary embodiments (Table 2), the variable vps_profile_tier_level_idx[i][j] denotes the index (in the list of profile_tier_level() syntax structures in the VPS) of the profile_tier_level() syntax structure that applies to level j of output level set i.

[0205] For SPS, Table 3 can be considered:

[0206] According to exemplary embodiments (Table 3), the variable pic_width_max_in_luma_samples denotes the maximum width in units of luma samples of each decoded picture with reference to the SPS. The variable pic_width_max_in_luma_samples must not be zero and must be an integer multiple of MinCbSizeY.

[0207] According to exemplary embodiments (Table 3), the variable pic_height_max_in_luma_samples denotes the maximum height in units of luma samples of each decoded picture with reference to the SPS. The variable pic_height_max_in_luma_samples must not be zero and must be an integer multiple of MinCbSizeY.

[0208] According to exemplary embodiments (Table 3), a sub_pics_present_flag variable equal to one indicates the presence of sub-picture parameters in the RBSP SPS syntax. The variable sub_pics_resent_flag equal to zero indicates that the sub-picture parameters are not present in the SPS RBSP syntax.

[0209] According to exemplary embodiments, when a bitstream is the result of a sub-bit stream extraction process and contains only a subset of sub-images of the input bit stream for the sub-bit stream extraction process, subpics_present_flag may need to be set to 1 in the RBSP SPS.

[0210] According to exemplary embodiments (Table 3), the variable sps_sub_pic_id_resent_flag equal to 1 means that sps_sub_pic_id[i] is present in the SPS. The sps_sub_pic_id_resent_flag variable equal to zero means that the sps_sub_pic_id[i] variable is not present in the SPS.

[0211] According to exemplary embodiments (Table 3), the variable sps_sub _pic_id_length_minus 1 denotes the number of bits used to display the sps_sub_pic_id[i][j] syntax element. sps_sub_pic_id_length_minus 1 must be between 0 and 15 inclusive. If this condition is not met, the value of sps_sub_pic_id+_ength_minus 1 is assumed to be Ceil(Log2(Max(2, sps_num_sub_pic_in_pic_minusl+1))) - 1.

[0212] According to exemplary embodiments (Table 3), the variable sps_sub_pic_id[i] denotes a sub-picture identifier i. The length of the sps_sub_pic_id[i] syntax element is sps_sub_pic_id_length_minus 1+1 bits. If this condition is not met, the value of sps_sub_pic_id[i] is assumed to be equal to i in the region from 0 to sps_num_sub_pic_in_pic_minus 1 inclusive for each i.

[0213] According to exemplary embodiments (Table 3), the variable sps_sub_pic_offset_x_in_luma_samples[i] denotes the horizontal offset (in units of luma samples) of the top left corner luma of subimage i relative to the top left corner luma of the composite image. If this condition is not met, the value of sps_sub_pic_offset_x_in_luma_samples[i] is assumed to be 0. The variable sps_sub_pic_offset_x_in_luma_samples[i] must be an integer multiple of the coding tree block size.

[0214] According to exemplary embodiments (Table 3), the variable sps_sub_pic_offset_y_in_luma_samples[i] denotes the vertical offset (in units of luma samples) of the upper left corner luma sample of subimage i relative to the upper left corner luma sample of the composite image. If this condition is not met, the value of sps_sub_pic_offset_y_in_luma_samples[i] is assumed to be 0. The variable sps_sub_pic_offset_y_in_luma_samples[i] must be an integer multiple of the coding tree block size.

[0215] According to exemplary embodiments (Table 3), the variable sps_sub_pic_width_in_luma_samples[i] denotes the width of subimage i in units of luma samples. The variable sps_sub_pic_width_in_luma_samples[i] must be an integer multiple of the coding tree block size.

[0216] According to exemplary embodiments (Table 3), the variable sps_sub_pic_height_in_luma_samples[i] denotes the height of subimage i in units of luma samples. The variable sps_sub_pic_height_in_luma_samples[i] must be an integer multiple of the coding tree block size.

[0217] According to exemplary embodiments (Table 3), the variable sps num output subpic sets minus 1 denotes the number of output subpicture sets in the encoded video sequence with reference to the SPS. If this condition is not met, sps num output layer sets minus 1 is assumed to be 0.

[0218] According to exemplary embodiments (Table 3), the variable sps_num_output_subpic_minus 1 [i] denotes the number of subimages in the output subimage set i.

[0219] According to exemplary embodiments (Table 3), the variable sps_sub_pic_id_oss[i][j] denotes a sub-picture identifier of output sub-picture j of sub-picture i. The length of the sps_sub_pic_id_oss[i][j] syntax element is sps_sub_pic_id_length_minus 1+1 bits. If this condition is not met, the value of sps_sub_pic_id_oss [i][j] is assumed to be] for each i in the region from 0 to sps_num_output_sub_pic_minus 1 [i] inclusive.

[0220] For PPS, Table 4 can be considered:

[0221] According to exemplary embodiments (Table 4), the variable sub_pics_updated_flag equal to one denotes an update of the sub-image layout data with syntax elements pointing to the updated sub-picture layout data in the PPS. The variable sub_pic_supdated_flag equal to 0 means that the sub-picture layout data is not updated.

[0222] According to exemplary embodiments (Table 4), a pps_sub_pic id_present_flag variable equal to one means that the pps_sub_pic_id[i] variable is present in the PPS. The variable sps_sub_pic_id_present_flag equal to zero means that pps_sub_pic_id[i] is not present in the PPS.

[0223] According to exemplary embodiments (Table 4), the variable pps_sub_pic_id_length_minus 1 plus 1 denotes the number of bits used to display the syntax element pps_sub_pic_id[i][j]. The value of pps_sub_pic_id_length_minu 1 must be between 0 and 15 inclusive. If this condition is not met, the value of pps_sub_pic_id_length_minusl is assumed to be Ceil(Log2(Max(2, pps_num_sub_pic_in_pic_minusl+1))) - 1.

[0224] According to exemplary embodiments (Table 4), the variable pps_sub_pic_id[i] denotes a sub-picture identifier i. The length of the syntax element pps_sub_pic_id[i] is sps_sub_pic_id_length_minusl+1 bits. If this condition is not met, the value of pps_sub_pic_id[i] is assumed to be equal to i in the region from 0 to pps_num_sub_pic_in_pic_minusl inclusive.

[0225] According to exemplary embodiments (Table 4), the variable pps_sub_pic_offset_x_in_luma_samples[i] denotes the horizontal offset (in units of luma samples) of the upper left corner luminance sample of subimage i relative to the upper left corner luminance sample of the composite image. If this condition is not met, pps_sub_pic_offset_x_in_luma_samples[i] is assumed to be 0. The variable pps_sub_pic_offset_x_in_luma_samples[i] must be an integer multiple of the coding tree block size.

[0226] According to exemplary embodiments (Table 4), the variable pps_sub_pic_offset_y_in_luma_samples[i] denotes the vertical offset (in units of luma samples) of the top left corner luma of subimage i relative to the top left corner luminance of the composite image. If this condition is not met, the value of sps sub_pic_offset_y in_luma_samples[i] is assumed to be 0. The variable sps_sub_pic_offset_y_in_luma_samples[i] must be an integer multiple of the coding tree block size.

[0227] According to exemplary embodiments (Table 4), the variable pps_sub_pic_width_in_luma_samples[i] denotes the width of subimage i in units of luma samples. The variable pps_sub_pic_width_in_luma_samples[i] must be an integer multiple of the coding tree block size.

[0228] According to exemplary embodiments (Table 4), the variable pps_sub_pic_height_in_luma_samples[i] denotes the height of subimage i in units of luma samples. The variable pps_sub_pic_height_in_luma_samples[i] must be an integer multiple of the coding tree block size.

[0229] According to exemplary embodiments (Table 4), the variable pps_num_output_subpic_minus [i] denotes the number of output sub-image sets in images with reference to PPS. If this condition is not met, the value pps_num_output_layer_sets_minus 1 is assumed to be 0.

[0230] According to exemplary embodiments (Table 4), the variable pps_num_outpu_ subpic_minus 1 [i] denotes the number of sub-images of the set of output sub-images i.

[0231] According to exemplary embodiments (Table 4), the variable pps_sub_pic_id_oss[i][j] denotes a sub-picture identifier of output sub-picture j of sub-picture i. The length of the syntax element pps_sub_pic_id_oss[i][j] is pps_sub_pic_id_length_minusl+1 bits. If this condition is not met, the value of pps_sub_pic_id_oss [i]fj] is assumed to be equal to j for each i in the region from 0 to pps_num_output_subpic_minus l[i] inclusive.

[0232] The above-described methods for signaling adaptive resolution parameters may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, in FIG. 7 shows a computer system (700) suitable for implementing some embodiments of the disclosed invention.

[0233] Computer software may be coded using any suitable machine code or computer language that may be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly, or through interpretation, execution of microcode, etc. .p., one or more computer central processing units (CPUs, central processing units), graphic processors (GPUs, Graphics Processing Units), etc.

[0234] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[0235] The computer system components (700) shown in FIG. 7 are illustrative in nature and are not intended to impose any limitation on the scope of application or functionality of computer software implementing embodiments of the present invention. The configuration of the components should also not be interpreted as having any dependency or requirement in connection with any one or combination of the components shown in the illustrative embodiment of the computer system (700).

[0236] The computer system (700) may include some human interface input devices. Such an input device may be responsible for input by one or more users through, for example, tactile input (e.g., keystrokes, waving, glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), olfactory input (not shown). Interface devices may also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, external sound), images (e.g. scanned images, photographic images received from a still camera), video (eg, 2D video, 3D video including stereoscopic video).

[0237] The human interface input devices may include one or more devices (one each of those depicted): keyboard (701), mouse (702), touch pad (703), touch screen (710), joystick (705), microphone (706), scanner (707), camera (708).

[0238] The computer system (700) may also include some human interface output devices. Such output devices may stimulate the senses of one or more users through, for example, tactile output, sound, light, and smell/taste. Such output devices may include tactile output devices (for example, haptic feedback via touch screen (710) or joystick (705), but may also be haptic feedback devices that do not serve as input devices), audio output devices (for example: speakers (709), headphones (not shown)), visual output devices (e.g., screens (710), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities , each with or without haptic feedback capabilities, some of which are capable of two-dimensional visual output or more than three-dimensional output through, for example, stereographic output; virtual reality goggles (not shown), holographic displays, and smoke tanks (not shown)) and printers (not shown).

[0239] The computer system (700) may also include human accessible storage devices and associated media, such as optical media including CD/DVD ROM/RW (720) with CD/DVD media (721), etc. etc., flash memory cards (720), removable hard drive or solid state drive (723), traditional magnetic media such as tape and floppy disk (not shown), dedicated ROM/ASIC/PLD based devices such as security hardware keys (not shown) and the like.

[0240] Those skilled in the art should also understand that the term "computer-readable media" as used in connection with the invention disclosed herein does not encompass transmission media, carrier waves, or other transient signals.

[0241] The computer system (700) may also include an interface to one or more communication networks. Communication networks can be, for example, wireless, wired, optical. Communication networks can be, in addition, local, global, urban, transport and industrial, real-time, delay, etc. Examples of communication networks include local area networks, such as Ethernet, wireless LANs, cellular networks, including GSM, 3G, 4G, 5G, LTE, etc., digital wide area networks of wired or wireless television, including cable television , satellite TV and terrestrial TV, transport and industrial include CANBus, etc. Some networks typically require external network interface adapters that are connected to some general purpose data ports or peripheral buses (749) (for example, computer system USB ports (700)); others are typically built into the core of the computer system (700) by connecting to a system bus as described below (eg, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (700) can communicate with other entities. Such communication may be unidirectional with receive-only capability (eg broadcast), unidirectional with transmit-only capability (eg CANbus to some CANbus devices), or bidirectional, eg to other computer systems using a local area network or a wide area digital network. Several protocols and protocol stacks may be used on each of these networks and the network interfaces described above.

[0242] The aforementioned human interface devices, human accessible storage devices, and network interfaces can be connected to the core (740) of the computer system (700).

[0243] The core (740) may include one or more central processing units (CPUs) (741), graphics processing units (GPUs) (742), specialized programmable processing modules in the form of Field Programmable Gate Areas (FPGAs). ) (743), hardware accelerators (744) for some tasks, etc. These devices, together with read-only memory (ROM) (745), random access memory (746), high-capacity internal storage such as user-inaccessible internal hard drives, SSD, etc. (747) can be connected via the system bus (748). In some computer systems, the system bus (748) may be available in the form of one or more physical connectors to allow for expansions with additional CPUs, GPUs, or the like. Peripherals can be connected either directly to the kernel system bus (748) or via a peripheral bus (749). Peripheral bus architectures include PCI, USB, and the like.

[0244] CPUs (741), GPUs (742), FPGAs (743), and accelerators (744) may execute some instructions that together may constitute the aforementioned computer code. This computer code may be stored in ROM (745) or RAM (746). Transient data may also be stored in RAM (746), while persistent data may be stored, for example, in an internal mass storage (747). Fast storage and retrieval from any storage device can be achieved through the use of cache memory, which can be closely associated with one or more CPU (741), GPU (742), mass storage (747), ROM (745), RAM (746) etc.

[0245] Computer-readable media may store computer code for performing various computer-implemented operations. The media and computer code may be specially created for the purposes of the present invention, or may be well known and available to those skilled in the art of computer software.

[0246] By way of example, and not limitation, a computer system having an architecture (700), and in particular a core (740), can provide functionality through execution by processor(s) (including(and) CPU, GPU, FPGAs, accelerators, etc.) software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with the user-accessible mass storage described above, as well as some persistent kernel storage (740), such as internal mass storage (747) or ROM (745). Software implementing various embodiments of the present invention may be stored on such devices and executed by the core (740). The computer-readable medium may include one or more storage devices or microcircuits, in accordance with specific needs. The software may direct the core (740) and, in particular, its processors (including CPUs, GPUs, FPGAs, etc.) to execute specific processes or specific parts of the specific processes described herein, including setting data structures stored in RAM (746) , and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality through logic hardwired or otherwise embodied in circuitry (e.g., an accelerator (744)) that may operate in place of or in conjunction with software to perform specific processes or specific portions of the specific processes described herein. A reference to software may cover logic, and vice versa when appropriate. Reference to computer-readable media may include a circuit (eg, an integrated circuit) that stores software to execute, a circuit embodying logic to execute, or both, as appropriate. The present invention encompasses any suitable combination of hardware and software.

[0247] Although several illustrative embodiments are described here, variations, permutations, and various replacement equivalents are possible and are within the scope of the invention. Thus, those skilled in the art can provide numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the invention and thus fall within its spirit and scope.

Claims (36)

1. A video encoding method performed by at least one processor, in which: receive video data; parsing the high-level syntax of the video data; determining whether the value of the high-level syntax element indicates a picture sequence number (ROS) value for an access unit (AU) of the video data; assigning at least one of the plurality of images, slices, and tiles of video data to an access block based on the value of the syntax element; determining whether the high-level syntax includes a flag indicating whether at least one of the images is divided into a plurality of sub-regions; and set the input image size of at least one of the images to the encoded image size signaled in the Video Sequence Parameter Set (SPS) in response to determining that the high-level syntax includes a flag indicating that at least one of the images is not divided into a set subdomains. 2. The method of claim. 1, in which the value of the syntax element indicates the number of consecutive elements of the plurality of images, slices and tiles of video data assigned to the access block. 3. The method of claim 1, wherein the high-level syntax is contained in a video data parameter set (VPS) and identifies a number of at least one type of video enhancement layer. 4. The method according to claim 1, which also includes: determining whether the high-level syntax contains a flag indicating whether the POC value is incremented uniformly per access block. 5. The method according to claim 4, which also includes: in response to determining that the VPS turns on a flag indicating uneven increase in the POC value per access unit, calculating the access unit number (AUC) based on the POC value and the image level value of the video data. 6. The method according to claim 4, which also includes: in response to determining that the VPS turns on a flag indicating a uniform increase in the POC value per access unit, calculating an access unit number (AUC) based on the POC value and the video sequence level value. 7. The method according to claim 1, which also includes: determining whether the SPS includes syntax elements signaling offsets corresponding to the video data level, in response to determining that the high-level syntax includes a flag indicating division of at least one of the images into a plurality of sub-regions. 8. The method of claim 7 wherein the offsets include a width offset and a height offset. 9. Device for video encoding, which includes: at least one memory configured to store computer program code; and at least one processor configured to access the computer program code and to operate according to the instructions of the computer program code, which includes: an acquisition code configured to cause at least one processor to acquire video data; parsing code configured to cause at least one processor to parse the high-level syntax; a determination code configured to cause at least one processor to determine whether the value of the high-level syntax element indicates a picture sequence number (POC) value for an access unit (AU) of video data; and setup code configured to cause at least one processor to assign at least one of a plurality of images, slices, and tiles of video data to an access block based on the value of the syntax element, wherein the determination code is also configured to cause at least one processor to determine whether the high-level syntax includes a flag indicating whether at least one of the images is divided into a plurality of sub-areas, and wherein the setting code is also configured to cause at least one processor, in response to determining that the high-level syntax includes a flag indicating that at least one of the images is not divided into a plurality of sub-regions, to set the size of at least one input image equal to the encoded image size signaled in the sequence parameter set (SPS) of the video data. 10. The apparatus of claim 9, wherein the value of the syntax element indicates the number of images, slices, and tiles of video data to be assigned to the access block. 11. The apparatus of claim 9, wherein the high-level syntax is contained in a video data parameter set (VPS) and identifies a number of at least one type of video enhancement layer. 12. The apparatus of claim 9, wherein the determination code is configured to cause at least one processor to determine whether the high-level syntax includes a flag indicating whether the POC value increases evenly per access unit. 13. The device according to claim 12, which also includes: calculation code configured to cause at least one processor, in response to determining that the VPS turns on a flag indicating uneven increase in the POC value per access unit, to calculate the access unit number (AUC) based on the POC value and the level value images of video data. 14. The device according to claim 12, which also includes: calculation code configured to cause at least one processor, in response to determining that the VPS turns on a flag indicating a uniform increase in the POC value per access unit, to calculate the access unit number (AUC) based on the POC value and the level value video data sequences. 15. The apparatus of claim 9, wherein the determination code is configured to cause at least one processor, in response to determining that the high-level syntax includes a flag indicating division of at least one of the images into a plurality of sub-regions, to determine, whether the SPS contains syntax elements signaling offsets corresponding to the video data layer. 16. A computer-readable storage medium that stores a program that, when executed by a computer, performs a computer video encoding method according to any one of paragraphs. 1-8.

Images