RU2787213C1 - Method for restricting referencing to a set of parameters in an encoded video stream - Google Patents

Abstract

FIELD: video data processing.

SUBSTANCE: group of inventions relates to video encoding and decoding, more specifically, according to the implementation examples, to referencing to a set of parameters and to boundaries thereof in an encoded video stream. Proposed is a method for video decoding executed by at least one processor. Method includes a stage of obtaining encoded video data containing the data of multiple semantically independent source pictures. Among the encoded video data, the fact whether the references are linked with any of the first access unit (AU) and the second AU is then determined according to at least one signal value of the picture order count (POC) included in the encoded video data. The first number of references set for the first AU and the second number of references set for the second AU are also output based on at least one signal POC value.

EFFECT: higher effectiveness of video encoding/decoding.

13 cl, 23 dwg, 1 tbl

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Application No. 62/954883, filed December 30, 2019, and U.S. Patent Application No. 17/063085, filed October 5, 2020, which are incorporated herein in their entirety. .

The technical field to which the invention belongs

[0002] The disclosed subject matter relates to video coding and decoding, and more specifically, according to embodiments, to a parameter set reference and its boundaries in an encoded video stream.

Prerequisites for the creation of the invention

[0003] Video encoding and decoding using motion-compensated inter-picture prediction has been known for decades. Uncompressed digital video may consist of a sequence of images, each image having a spatial size of, for example, 1920×1080 luminance samples and their associated chrominance samples. An image sequence can have a fixed or variable image rate (informally also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (resolution of 1920x1080 luminance samples at 60Hz frame rate) requires bandwidth close to 1.5Gbps. An hour of such video requires more than 600 GB of storage space.

[0004] One of the goals of video encoding and decoding may be to reduce the redundancy of the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements by two orders of magnitude or more in some cases. Both lossless and lossy compression can be used, as well as a combination of both. Lossless compression refers to technologies where an exact copy of the original signal can be reconstructed from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely used. The amount of distortion allowed depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television delivery applications. The achievable compression ratio may reflect the following: higher acceptable/acceptable distortion may result in higher compression ratios.

[0005] The video encoder and decoder may use technologies from several broad categories, including, for example, motion compensation, transformation, quantization, and entropy coding, some of which will be introduced below.

[0006] Historically, video encoders and decoders tended to work with a given image size, which in most cases was determined and remained constant for a coded video sequence (CVS, Coded Video Sequence), a group of pictures (GOP, Group of Pictures) or a similar time interval from multiple images. For example, the MPEG-2 Moving Picture Experts Group (MPEG) standard knows system structures for changing horizontal resolution (and hence picture size) depending on factors such as scene activity, but only for I images. (Intra-coded picture - the image is encoded using only the information that is contained in itself), therefore, usually for a group of images (GOP, Group of Pictures). Reference picture resampling to use different resolutions in a Coded Video Sequence (CVS) is known, for example, from Annex P of Recommendation H.263 of the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T). However, here the image size is not changed, only the reference images are forward-sampled, potentially using only parts of the image canvas (in the case of downsampling) or only parts of the captured scene (in the case of upsampling). In addition, Annex Q of Recommendation H.263 allows an individual macroblock to be resampled by a factor of two (in each dimension) up or down. Again, the image size remains the same. The macroblock size is fixed in H.263 and therefore does not need to be signaled.

[0007] Image resizing in predictive images has become a more common trend in modern video coding. For example, the VP9 standard (Video Processor 9 - Video Processor 9) allows you to resample the reference image and change the resolution for the entire image. Similarly, some of the proposals made regarding Versatile Video Coding (VVC) (including, for example, Hendry, et. al, "On adaptive resolution change (ARC) for VVC", document JVET-M0135-vl, 9 -January 19, 2019, JVET (Joint Video Experts Team) within the joint program (JVT, Joint Video Team, fully incorporated in this document) allows resampling of entire reference pictures to obtain a different, higher or lower , permissions. This document proposes to encode the various possible resolutions in a sequence parameter set and refer to them using syntax elements for each image in the image parameter set.

[0008] Recently, some attention has been drawn to the aggregation of a compressed area, or the extraction of multiple semantically independent parts of an image into a single video image. In particular, in the context of e.g. 360-degree encoding or certain surveillance applications, multiple semantically independent source images (e.g., six cube surfaces of a 360-degree scene projected onto a cube, or individual camera inputs in the case of a multi-camera surveillance setup) may require separate settings. adaptive resolution to cope with the different activity of each scene at a given moment in time.

The essence of the invention

[0009] A technology for signaling adaptive image size in a video bitstream is disclosed.

[0010] It includes a method and apparatus comprising a memory configured to store machine program code and a processor or processors configured to access the machine program code and operate in accordance with instructions of the machine program code. The computer program includes an acquisition code configured to cause at least one processor to acquire video data containing multiple semantically independent source image data; determination code configured to cause at least one processor to determine among the video data whether links are associated with any of the first access unit (AU, Access Unit) and the second AU according to at least one signal value of the sequence number of the picture (POC, Picture Order Count) included in the video data; and an output code configured to cause the at least one processor to output the first link number set for the first AU and the second link number set for the second AU based on the at least one POC signal value.

[0011] According to exemplary embodiments of the invention, the links contain at least one of images, slices, and tiles of video data.

[0012] According to embodiments of the invention, determining if the links are associated with either of the first AU and the second AU includes comparing the respective POC values of each of the links with at least one POC signal value.

[0013] According to embodiments of the invention, determining if the links are associated with either of the first AU and the second AU further includes setting the first number of links for the first AU in response to determining that each of the first number of links, respectively, contain one of a plurality of values POC less than at least one POC signal value, and setting a second link number for the second AU in response to determining that each of the second number of links, respectively, contains one of the second set of POC values equal to or greater than at least one ROC signal value.

[0014] According to exemplary embodiments of the invention, the links comprise slices, and at least one POC signal value is included in the slice header of the video data.

[0015] According to exemplary embodiments of the invention, the video data contains Video Parameter Set (VPS) data identifying a plurality of spatial levels of video data.

[0016] According to embodiments of the invention, at least one POC signal value is included in a Video Parameter Set (VPS) of video data.

[0017] According to embodiments of the invention, the determination code is further configured to cause at least one processor to determine whether the VPS data contains at least one flag indicating whether one or more links are divided into a plurality of sub-areas, and determine if when the at least one flag indicates that the one or more links are divided into a plurality of sub-regions of at least one of the full image width and the full image height in image brightness samples of the one or more links.

[0018] According to embodiments of the invention, the determination code is further configured to cause at least one processor to determine, in the event that at least one flag indicates that one or more links are divided into a plurality of sub-areas, the signaling value included in the sequence parameter set video data, which determines the offset of a part of at least one of the sub-regions.

[0019] According to embodiments of the invention, a plurality of semantically independent source images represent a spherical 360-degree image.

Brief description of the drawings

[0020] Additional features, essence and various advantages of the disclosed subject matter will be better understood from the following detailed description and the accompanying drawings, in which:

[0021] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system in accordance with an embodiment of the invention.

[0022] FIG. 2 is a schematic diagram of a simplified block diagram of a communication system in accordance with an embodiment of the invention.

[0023] FIG. 3 is a schematic diagram of a simplified block diagram of a decoder in accordance with an embodiment of the invention.

[0024] FIG. 4 is a schematic diagram of a simplified block diagram of an encoder in accordance with an embodiment of the invention.

[0025] FIG. 5A and 5B are schematic diagrams of embodiments of Adaptive Resolution Change (ARC) signaling in accordance with the prior art or embodiment as indicated.

[0026] FIG. 6 is an example of a syntax table according to an embodiment of the invention.

[0027] FIG. 7 is a schematic representation of a computer system in accordance with an embodiment of the invention.

[0028] FIG. 8 is an example of a prediction structure for adaptive resolution scalability.

[0029] FIG. 9 is an example of a syntax table according to an embodiment of the invention.

[0030] FIG. 10 is a schematic diagram of a simplified flowchart of a Picture Sequence Number (ROS) parsing and decoding cycle per access block and access block number value.

[0031] FIG. 11 is a schematic diagram of a video bitstream structure containing layered sub-images.

[0032] FIG. 12 is a schematic illustration of a display of a selected sub-image with enhanced resolution.

[0033] FIG. 13 is a flowchart of a process for decoding and displaying a video bitstream containing layered sub-images.

[0034] FIG. 14 is a schematic illustration of a 360° video display with sub-image enhancement level.

[0035] FIG. 15 is an example of sub-image arrangement information and their respective level structure and image prediction.

[0036] FIG. 16 is an example of sub-image layout information and their respective layer structure and image prediction with local area spatial scalability modality.

[0037] FIG. 17 is an example of a syntax table for sub-image layout information.

[0038] FIG. 18 is an example of a Supplemental Enhancement Information (SEI) message syntax table for sub-image layout information.

[0039] FIG. 19 is an example of a syntax table for specifying output levels and profile/tier/level information for each set of output levels.

[0040] FIG. 20 is an example of a syntax table for specifying an output level mode for each set of output levels.

[0041] FIG. 21 is an example of a syntax table for specifying the current sub-image of each level for each set of output levels.

[0042] FIG. 22 is an example of a parameter set reference in a non-reference level.

Problem to be solved

[0043] Since aggregating a compressed region into a single video image or extracting multiple semantically independent portions of an image has received some attention in the context of, for example, 360-degree encoding or certain video surveillance applications, multiple semantically independent source images (for example, six faces of a cube projected onto a 360-degree cube) degree scene or individual camera inputs in the case of a multi-camera installation) may require separate adaptive resolution settings to account for the different activity of each scene at a given time. Therefore, this document, inter alia, discloses encoders that at a given time can choose to use different resampling factors for different semantically independent images that make up an entire 360-degree or video surveillance scene. When combined into a single picture, this in turn requires that the reference picture be resampled and that there be adaptive resolution coding signaling for portions of the encoded picture.

Detailed description

[0044] FIG. 1 illustrates a simplified block diagram of a communication system 100 according to one embodiment of the present invention. System 100 may include at least two terminals 110 and 120 interconnected via network 150. For unidirectional data transmission, first terminal 110 may encode video data at a local location for transmission to another terminal 120 via network 150. Second terminal 120 may receive encoded video data of another terminal from the network 150, decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in multimedia service applications and the like.

[0045] FIG. 1 shows a second pair of terminals 130, 140 providing support for bidirectional encoded video transmission, such as may occur during a video conference. For bi-directional data transmission, each terminal 130, 140 may encode video data captured at a local location for transmission to another terminal via network 150. Each terminal 130, 140 may also receive encoded video data transmitted by another terminal, may decode the encoded data, and may display the reconstructed video data. on the local display device.

[0046] FIG. 1, terminals 110, 120, 130, 140 may be shown as servers, personal computers, and smartphones, but the principles of the present disclosure cannot be limited to this. Embodiments of the present invention find application in laptop computers, tablet computers, media players and/or specialized video conferencing equipment. Network 150 represents any number of networks that transmit encoded video data between terminals 110, 120, 130, 140, including, for example, wired and/or wireless networks. Communication network 150 may communicate over circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks and/or the Internet. For the purposes of this discussion, the architecture and topology of network 150 may not be essential to the operation of this disclosure, unless explained herein below.

[0047] FIG. 2 shows, as an example of application for the disclosed subject matter, the placement of a video encoder and decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications, including, for example, video conferencing, digital television, compressed video storage on digital media, including CD, Compact Disk, DVD, Digital Versatile Disk), memory card, etc.

[0048] The streaming system may include a video capture subsystem 213, which may include a video source 201, such as a digital camera, creating, for example, a stream 202 of uncompressed video samples. This sample stream 202, depicted in bold to emphasize the large amount of data compared to the encoded video bitstreams, may be processed by an encoder 203 connected to camera 201. Encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream 204, shown as a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server 205 for future use. One or more streaming clients 206, 208 may access the streaming server 205 to obtain copies 207, 209 of the encoded video bitstream 204. Client 206 may include a video decoder 210 that decodes an incoming copy 207 of an encoded video bitstream and produces an outgoing video sample stream 211 that can be displayed on a display 212 or other rendering device (not shown). In some streaming systems, video bitstreams 204, 207, 209 may be encoded according to certain video coding/compression standards. Examples of these standards include ITU-T Recommendation H.265. A video coding standard known informally as Versatile Video Coding (VVC) is under development. The disclosed subject matter may be used in the context of the VVC standard.

[0049] FIG. 3 may represent a functional block diagram of a video decoder 210 according to one embodiment of the present invention.

[0050] Receiver 310 may receive one or more encoded video sequences to be decoded by decoder 210; in the same or another embodiment of the invention, one coded video sequence at a time, where the decoding of each coded video sequence is independent of other coded video sequences. An encoded video sequence may be received from channel 312, which may be a hardware/software connection to a storage device that stores the encoded video data. Receiver 310 may receive encoded video data along with other data, such as encoded audio data and/or ancillary data streams, which may be forwarded to their respective uses (not shown). The receiver 310 may separate the encoded video sequence from other data. To combat network jitter, a buffer memory 315 may be included between the receiver 310 and the entropy decoder/parser 320 (hereinafter referred to as the "parser"). When receiver 310 receives data from a storage/transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, buffer 315 may not be needed or may be small. For use in packet networks with maximum efficiency, such as the Internet, buffer 315 may be required, it may be relatively large, and preferably may have an adaptive size.

[0051] Video decoder 210 may include a parser 320 for recovering symbols 321 from an entropy coded video sequence. The categories of these symbols include information used to control the operation of the decoder 210 and potentially information to control a visualization device such as a display 212 that is not an integral part of the decoder but can be attached to it as shown in FIG. 2. The control information for the rendering device(s) may be in the form of Supplementary Enhancement Information (SEI) or Video Usability Information (VUI) parameter set fragments (not shown). Parser 320 may parse/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be in accordance with a video coding technology or standard, including variable length coding, Huffman coding, arithmetic coding with or without context, etc. Parser 320 may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOPs), images, tiles, slices, macroblocks, code blocks (CU, Coding Unit), blocks (blocks), transformation units (TU, Transform Unit), prediction units (PU, Prediction Unit, ) etc. The entropy decoder/parser may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0052] Parser 320 may perform an entropy decoding/parsing operation on the video sequence received from buffer 315 so as to produce symbols 321.

[0053] Symbol recovery 321 may include several different blocks depending on the type of encoded video image or portions thereof (eg, intra-inter prediction encoded image, intra-inter prediction encoded block) and other factors. Which blocks to use and how can be controlled by the subgroup control information that has been parsed from the encoded video sequence by the parser 320. The flow of such subgroup control information between the parser 320 and the plurality of blocks is not shown for clarity.

[0054] In addition to the functional blocks already mentioned, the decoder 210 can be conceptually subdivided into a number of functional blocks, as described below. In a practical implementation operating under commercial constraints, many of these blocks interact closely with each other and can, at least in part, be integrated into each other. However, for purposes of describing the disclosed subject matter, the conceptual division into functional blocks below is appropriate.

[0055] The first block is the scaling/inverse transform block 351. The scaling/inverse transform block 351 receives a quantized transform coefficient as well as control information including which transform to use, block size, quantization factor, quantization scaling matrices, and so on. as character(s) 321 from the parser 320. It may output blocks containing sample values that may be entered into the aggregator 355.

[0056] In some cases, the output samples of the scaler/inverse transform block 351 may refer to an intra-prediction-encoded block, that is: a block that does not use prediction information from previously restored images, but can use prediction information from previously restored parts of the current image . Such prediction information may be provided by intra prediction block 352 . In some cases, the intra frame prediction block 352 generates a block of the same size and shape as the reconstructed block, using the surrounding already restored information extracted from the current (partially restored) image 356. The aggregator 355 in some cases adds, for each sample, the prediction information generated by the block 352 intra-frame prediction, to the information of the output samples provided by the block 351 scaling/inverse transformation.

[0057] In other cases, the output samples of the scaler/inverse transform block 351 may refer to a block encoded with inter prediction and possibly with motion compensation. In such a case, the motion compensated predictor 353 may access the reference picture memory 357 to fetch samples used for prediction. After compensating for movement of the selected samples according to the block related symbols 321, these samples can be added by an aggregator 355 to the output of the scaler/demap block (in this case referred to as residual samples or residual signal) to form output sample information. Reference picture memory addresses from where the motion compensator selects prediction samples may be controlled by motion vectors available to the motion compensator in the form of symbols 321, which may have X, Y, and reference picture components, for example. Motion compensation may also include interpolation of sample values retrieved from reference picture memory when accurate subsample motion vectors, motion vector prediction mechanisms, and so on are used.

[0058] The output samples of the aggregator 355 may be subjected to various loop filtering techniques in the loop filter block 356 . Video compression techniques may include in-loop filtering techniques that are controlled by parameters included in the encoded video bitstream and provided to loop filter block 356 as symbols 321 from parser 320, but may also be responsive to meta-information obtained during decoding of previous (in decoding order) of parts of an encoded image or encoded video sequence, as well as in response to previously recovered and loop-filtered sample values.

[0059] The output of loop filter block 354 may be a stream of samples that may be output to renderer 212 as well as stored in reference picture memory for use in future inter-picture prediction.

[0060] Some encoded pictures after full recovery can be used as reference pictures for future prediction. Once the encoded picture has been completely reconstructed and the encoded picture has been identified as a reference picture (e.g., by the parser 320), the current reference picture 356 may become part of the reference picture buffer 357, and fresh current picture memory may be reallocated before the next encoded picture is decommissioned.

[0061] Video decoder 210 may perform decoding operations according to a predetermined video compression technology that may be documented in a standard such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense that it adheres to the syntax of the video compression technology or standard as specified in the video compression technology document or standard and, in particular, in the profiles documented therein. Compliance also requires that the complexity of the encoded video sequence be within the limits defined by the level of technology or video compression standard. In some cases, the levels limit the maximum image size, the maximum frame rate, the maximum recovery sample rate (measured in Ms/s, for example), the maximum reference image size, and so on. The limits set by the layers may in some cases be further limited by Hypothetical Reference Decoder (HRD) specifications and metadata for managing the HRD buffer signaled in the encoded video sequence.

[0062] In one form of the invention, the receiver 310 may receive additional (redundant) data with the encoded video. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by video decoder 210 to properly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement levels, redundant slices, redundant images, forward error correction codes, and so on.

[0063] FIG. 4 may be a functional block diagram of a video encoder 203 according to one embodiment of the present invention.

[0064] The encoder 203 may receive video samples from a video source 201 (not part of the encoder) that may capture video image(s) for encoding by the encoder 203.

[0065] Video source 201 may provide the original video sequence to be encoded by encoder 203 in the form of a digital video sample stream that may have some suitable bit depth (eg: 8 bits, 10 bits, 12 bits,... ), some color space (for example, Y CrCB space as recommended by ITU-R BT.601, RGB, ...), some suitable sampling structure (for example, Y CrCb 4:2:0, Y CrCb 4:4: 4. In a media serving system, video source 201 may be a storage device storing pre-prepared video.In a videoconferencing system, video signals, source 203 may be a camera that captures local image information as a video sequence. The images themselves can be organized as a spatial array of pixels, with each pixel containing one or more samples. etov depending on the used sample structure, color space, etc. Those skilled in the art can easily understand the relationship between pixels and samples. The description below is based on readings.

[0066] According to one embodiment of the invention, encoder 203 may encode and compress images of the original video sequence into encoded video sequence 443 in real time or with some other time constraints as required by the application. Providing an appropriate code rate is one of the functions of the controller 450. The controller controls other functional blocks, as described below, and is operatively connected to these blocks. The connection is not shown for clarity. Parameters set by the controller may include parameters related to rate control (picture skip, quantizer, lambda value (weighting factor) for rate-distortion optimization technologies, ...), image size, group of pictures (GOP) layout, maximum search range motion vectors, etc. Those skilled in the art can readily identify other functions of controller 450 as they may relate to video encoder 203 optimized for a particular system design.

[0067] Some video encoders operate using what those skilled in the art know as a "coding loop". As a simplified description, an encoding loop may consist of an encoding part of an encoder 430 (hereinafter "source encoder") (responsible for creating symbols based on the input image to be encoded and the reference image(s)), and a (local) decoder 433 , built into the encoder 203, which decompresses the symbols to create sample data that the (remote) decoder can also create (because any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies discussed in the disclosed subject matter). This reconstructed sample stream is entered into the reference picture memory 434 . Since the decoding of a symbol stream results in bit-accurate results regardless of the location of the decoder (local or remote), the contents of the reference picture buffer also have bit-accuracy between the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" using prediction during decoding. This is the principle of reference picture synchronism (and the resulting drift if synchronism cannot be maintained, for example due to channel errors). This fundamental principle of reference picture synchronism (and the resulting drift if synchronism cannot be maintained due to, for example, channel errors) is well known to those skilled in the art.

[0068] The operation of the "local" decoder 433 may be the same as that of the "remote" decoder 210, which has already been described in detail above in connection with FIG. 3. Referring also briefly to FIG. 4, however, since the symbols are available and the entropy encoder 445 and parser 320 can be losslessly encoded/decoded into the encoded video sequence, the entropy decoding portions of decoder 210, including channel 312, receiver 310, buffer 315, and parser 320 may not be fully implemented in the local 433 decoder.

[0069] At this point, it can be observed that any decoding technology other than parsing/entropy decoding that is present in a decoder must also necessarily be present in a substantially identical functional form in a corresponding encoder. For this reason, the disclosed subject matter focuses on the operation of the decoder. The description of encoding technologies can be abbreviated as they are the inverse of fully described decoding technologies. Only in some areas a more detailed description is required, which is given below.

[0070] As part of its operation, source encoder 430 may perform motion-compensated predictive encoding that predictively encodes an input frame with reference to one or more previously encoded frames from the video sequence, which have been designated "reference frames". Thus, the encoding engine 432 encodes the differences between the pixel blocks of the input frame and the pixel blocks of the reference frame(s) that can be selected as prediction references for the input frame.

[0071] The local video decoder 433 may decode the encoded video data of the frames, which can be marked as reference frames, based on the symbols generated by the source encoder 430. The operations of the encoding engine 432 may be predominantly lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence may typically be a copy of the original video sequence with some errors. in the reference picture cache 434. Thus, the encoder 203 can locally store copies of the reconstructed reference frames that have the same content as the reconstructed reference frames to be received by the far-end video decoder (in the absence of transmission errors).

[0072] Predictor 435 may perform predictive searches for encoding engine 432 . That is, for a new frame to be encoded, the predictor 435 may search the reference picture memory 434 for sample data (as candidate pixel reference blocks) or certain metadata such as reference picture motion vectors, block shapes, etc., this can serve as a suitable reference for prediction for new images. Predictor 435 may operate on a block-per-pixel basis to find appropriate references for prediction. In some cases, as determined by the search results obtained by the predictor 435, the input image may have prediction references taken from a plurality of reference pictures stored in the reference picture memory 434 .

[0073] Controller 450 may control the encoding operations of video encoder 430, including, for example, setting parameters and subgroup parameters used to encode video data.

[0074] The output of all the above functional blocks may be entropy encoded in an entropy encoder 445. The entropy encoder translates symbols generated by different functional blocks into an encoded video sequence by lossless symbol compression according to various techniques, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0075] The transmitter 440 may buffer the encoded video sequence(s) generated by the entropy encoder 445 to prepare it for transmission over a link 460, which may be a hardware/software link to a data storage device that will store the encoded video data. Transmitter 440 may combine the encoded video data from video encoder 430 with other data to be transmitted, such as encoded audio data and/or ancillary data streams (sources not shown).

[0076] The controller 450 may control the operation of the encoder 203. During encoding, the controller 450 may assign to each encoded picture a certain type of encoded picture, which can affect the encoding technologies that can be applied to the corresponding picture. For example, images can often be assigned as one of the following frame types:

[0077] An intra picture (I picture, Intra Picture) may be one that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow the use of various intra-encoded types, including, for example, updated images, using an independent decoder. Variants of I-pictures and their respective uses and features may be used.

[0078] A predictively encoded picture (P picture) (P picture, Predictive picture) may be one that can be encoded and decoded using intra prediction or inter prediction using at most one motion vector and a reference index for prediction sample values of each block.

[0079] A picture encoded with bidirectional prediction (B-picture) (In Picture, Bi-directionally predictive picture) may be one that can be encoded and decoded using intra prediction or inter prediction using no more than two motion vectors and reference indexes for predicting the values of the samples of each block. Similarly, multi-prediction encoded pictures may use more than two reference pictures and their associated metadata to reconstruct a single block.

[0080] Source images can typically be subdivided spatially into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. The blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the coding assignment applied to the corresponding block images. For example, I-picture blocks may be predictively encoded or they may be predictively encoded with reference to already encoded blocks of the same picture (spatial prediction or intra prediction). P-picture pixel blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one previously encoded reference picture. Blocks of B-pictures may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one or two previously encoded reference pictures.

[0081] The video encoder 203 may perform encoding operations according to a predetermined video coding technology or standard, such as ITU-T Recommendation H.265. During operation, video encoder 203 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore follow the syntax given by the video coding technology or standard being used.

[0082] In one form of the invention, the transmitter 440 may transmit additional data with encoded video. Video encoder 430 may include such data as part of an encoded video sequence. The additional data may comprise temporal/spatial/SNR enhancement levels, other forms of redundant data such as redundant images and slices, additional enhancement information (SEI) messages, video usage information (VUI) parameter set fragments, and so on.

[0083] Before a more detailed description of certain aspects of the disclosed subject matter, it is necessary to introduce a few terms that will be referred to in the remainder of this description.

[0084] Hereinafter, the term "sub-image" refers, in some cases, to a rectangular structure of samples, blocks, macroblocks, code blocks, or similar objects that are semantically grouped and that can be independently encoded with a changed resolution. One or more sub-images may form an image. One or more encoded sub-images may form an encoded image. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In certain environments, one or more encoded sub-images may be assembled in the compressed region, without transcoding to the sample level, into the encoded image, and in the same or some other cases, one or more encoded sub-images may be extracted from the encoded image in the compressed region.

[0085] Adaptive resolution change (ARC) refers to mechanisms that allow the resolution of a picture or sub-picture in an encoded video sequence to be changed, for example, by resampling a reference picture. The ARC parameters may refer to the types of control information needed to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, output and/or reference picture resolutions, various control flags, and so on.

[0086] The above description focuses on encoding and decoding a single semantically independent encoded video image. Before describing the implications of encoding/decoding multiple sub-images with independent ARC parameters and its expected additional complexity, an option for ARC parameter signaling should be described.

[0087] FIG. 5 shows several new options for ARC parameter signaling. As noted for each of the options, they have certain advantages and certain disadvantages in terms of coding efficiency, complexity and architecture. The video coding standard or technology may choose one or more of these options or other options for signaling the ARC parameters. The options may not be mutually exclusive and may possibly be interchanged depending on the needs of the application, the standards used, or the choice of encoder.

[0088] ARC parameter classes may include:

- upsampling/downsampling factors, either single or combined across the X and Y dimensions;

- upsampling/downsampling factors, with the addition of a temporal dimension, indicating upscaling/downscaling at a constant rate for a given number of images.

- any of the above two may include encoding one or more supposedly short syntax elements, which may indicate a table containing the factor(s).

- resolution in the X or Y dimension in units of samples, blocks, macroblocks, CUs, or some other suitable granularity of the input picture, output picture, reference picture, coded picture, combined or alone. If there is more than one resolution (eg, one for the input image, one for the reference image), then in some cases one set of values may be inferred from another set of values. This can be controlled, for example, using flags. See below for a more detailed example.

- "warp" coordinates like those used in H.263 Annex P, again with the appropriate granularity as described above. Annex P H.263 defines one efficient way to encode such strain coordinates, but other potentially more efficient ways may also be developed. For example, the reversible "Huffman" style variable length encoding for warp coordinates according to Annex P can be replaced by a suitable length binary encoding, where the length of the binary codeword can be, for example, obtained from the maximum image size, perhaps multiplied by a certain factor and shifted by a certain value to allow "warping" beyond the bounds of the maximum image size.

- filter options with upsampling or downsampling. In the simplest case, there can only be one filter for upsampling and/or downsampling. However, in some cases it may be beneficial to provide more flexibility in the filter structure, and this may require filter parameter signaling. Such parameters may be selected by an index into a list of possible filter structures, a filter may be fully specified (e.g., by a list of filter coefficients using suitable entropy coding techniques), a filter may be implicitly selected by upsampling/downsampling relationships that, in turn are signaled according to one of the mechanisms mentioned above, and so on.

[0089] The following description assumes encoding a finite set of upsampling/downsampling coefficients (the same coefficient must be used for both the X dimension and the Y dimension) indicated by a codeword. This codeword may advantageously be encoded with variable length, for example, using the exponential Golomb code (Ext-Golomb) common to certain syntax elements in video coding specifications such as H.264 and H.265. One suitable mapping of values to upsampling/downsampling factors may, for example, correspond to the following table.

[0090] Many similar mappings can be developed in accordance with the needs of the application and the capabilities of the upscaling and downscaling mechanisms available in the video compression technology or standard. The table can be expanded to more values. Values may also be represented by entropy encoding mechanisms other than Exponential Golomb codes, such as using binary encoding. This may have certain advantages when the oversampling factors were of interest outside of the video processing mechanisms themselves (primarily the encoder and decoder), for example, through Media Aware Network Elements (MANE). It should be noted that for the (presumably) most common case where no resolution change is required, a short exponential Golomb code can be chosen; in the table above, only one bit. This may have an advantage in coding efficiency over using binary codes for the most common case.

[0091] The number of entries in the table, as well as their semantics, can be fully or partially configurable. For example, the basic table structure may be conveyed in a "high level" parameter set, such as a sequence or decoder parameter set. Alternatively, or in addition, one or more such tables may be defined in a video coding technology or standard and may be selected, for example, by a decoder or sequence parameter set.

[0092] Next, we will describe how the upsampling/downsampling ratio (ARC information) encoded as described above can be included in the syntax of a video coding technology or standard. Similar considerations may apply to one or more codewords driving the upsampling/downsampling filters. See the discussion below for when a filter or other data structure requires relatively large amounts of data.

[0093] The H.263 Annex P shown in Figure 500A includes the ARC information 502 in the form of four warp coordinates in the image header 501, specifically in the H.263 standard PLUSPTYPE header 503 extension. This may be a reasonable design choice when a) there is an image header available and b) frequent changes to the ARC information are expected. However, the amount of overhead when using H.263 style signaling can be quite high, and the scaling factors may not apply to picture boundaries because the picture header may be variable.

[0094] The JVCET-M135-vl document mentioned above includes ARC reference information 505 (index) located in the image parameter set 504, an index table 506 including target resolutions, which in turn is located within the sequence parameter set 507. Placing a candidate resolution in table 506 in sequence parameter set 507, in accordance with the verbal statements made by the authors, may be justified by using the SPS (Sequence Parameter Set) as a compatibility negotiation point during capability exchange. The resolution may vary within the limits set by the values in table 506 from image to image by referring to an appropriate set 504 of image parameters.

[0095] Still referring to FIG. 5, the following additional options may exist for transmitting ARC information in a video bitstream. Each of these options has certain advantages over the current state of the art, as described above. The options may be present in the same video coding technology or standard at the same time.

[0096] In an embodiment, ARC information 509 as in illustration 500B, such as a resampling factor (scaling), may be present in a slice header, a GOB header, a tile header, or a tile group header 508 (hereinafter referred to as the tile group header). This may be sufficient for small size ARC information such as a single variable length codeword ue(v) or a fixed length codeword of multiple bits, such as shown above. Having the ARC information in the tile group header directly has the additional benefit that the ARC information may be applicable to a sub-image represented by that tile group, for example, rather than to the entire image. See also below. In addition, even if a video compression technology or standard provides only adaptive resizing of the entire image (as opposed to, for example, adaptive resizing based on a group of tiles), placing it in an H.263-style image header has certain advantages in terms of ensure fault tolerance.

[0097] In the same or another embodiment of the invention, the ARC information 512 itself may be present in a corresponding parameter set 511, such as, for example, a picture parameter set, a header parameter set, a tile parameter set, an adaptation parameter set, and so on. (a set of adaptation parameters is shown). The scope of this set of parameters may advantageously be no larger than an image, eg represent a group of tiles. The use of ARC information is implicitly done by activating the appropriate set of options. For example, when a video coding technology or standard considers only image-based ARC, then a set of image parameters or its equivalent may be appropriate.

[0098] In the same or another embodiment of the invention, ARC reference information 513 may be present in a tile group header 514 or a similar data structure. This reference information 513 may refer to a subset of the ARC information 515 available in a parameter set 516 with a scope larger than a single picture, such as a sequence parameter set or a decoder parameter set.

[0099] An additional level of indirection, involving the activation of a Picture Parameter Set (PPS, Picture Parameter Set) from the tile element group header, PPS, SPS sets, as used in JVET-M0135-vl, seems unnecessary, since picture parameter sets, also , as sequence parameter sets, can be used (and are used in some standards such as RFC3984) to negotiate capabilities or declarations. If, however, the ARC information is to be applicable to a sub-image represented, for example, also by tile groups, then a parameter set with an activation area limited by the tile group, such as an adaptation parameter set or a header parameter set, may be a better choice. Also, if the ARC information is of more than a small size—for example, contains filter control information such as a set of filter coefficients—then the parameter may be a better choice than using header 508 directly from a coding efficiency point of view, as these settings may be suitable. for reuse by future images or subimages by reference to the same set of parameters.

[0100] When using a sequence parameter set or other higher level parameter set with a scope spanning multiple images, certain considerations may apply:

1. The parameter set for storing the ARC information table 516 may in some cases be a sequence parameter set, but in other cases preferably a decoder parameter set. The decoder parameter set may have an activation area of several CVS coded video sequences, namely the coded video stream, i. e. all bits of the encoded video from session start to session break. Such scope may be more appropriate because possible ARC factors may be a decoder function, perhaps implemented in hardware, and hardware functions generally do not change with any CVS sequence (which in at least some entertainment systems is group of images one second or less in length). However, the inclusion of a table in the sequence parameter set is explicitly included in the placement options described here, in particular in connection with point 2 below.

2. The ARC reference information 513 can advantageously be placed directly in the image/tile slice/group of blocks (GOB, Group Of Blocks)/tile group (hereinafter referred to as the tile group header) header 514 rather than in the image parameter set as in the JVCET document. -M0135-vl. The reason is that when an encoder wants to change one value in a picture parameter set, such as ARC help information, for example, it must create a new PPS set and refer to this new PPS set. Assume that only the ARC reference information is changed, but other information, such as quantization matrix information in the PPS set, for example, remains. Such information may be of considerable size and will need to be retransmitted in order to complete a new PPS set. Since the ARC reference information may be a single codeword, such as an index in table 513, and this will be the only value that changes, it would be cumbersome and wasteful to retransmit the entirety, including, for example, the quantization matrix information. In this regard, it may be significantly better in terms of coding efficiency to avoid indirection through the PPS set, as proposed in JVET-M0135-vl. Similarly, placing the ARC reference information in the PPS has the additional disadvantage that the ARC information referred to in the ARC reference 513 must necessarily apply to the entire image and not to a sub-image, since the task of activating the image parameter set is the image. .

[0101] In the same or another embodiment of the invention, ARC parameter signaling may follow a detailed example as shown in FIG. 6. FIG. 6 is a diagram showing examples of syntax in the representation used in video coding standards since at least 1993. The presentation of such diagrams showing examples of syntax is roughly equivalent to C-style programming. Lines in bold indicate syntax elements present in the bitstream, lines without bold type often indicate control flow or variable settings.

[0102] A tile group header 601, as an example of a header syntax structure applicable to a (possibly rectangular) portion of an image, may conditionally contain a variable length syntax element encoded with exponential Golomb code, dec_pic_size_idx 602 (in bold). The presence of this syntax element in a tilegroup header can be triggered by using adaptive resolution 603 - here the flag value is not in bold, meaning that the flag is present in the bitstream at the point where it occurs in the syntax diagram. Whether adaptive resolution is used for this image or parts of it can be signaled in some high-level syntactic structure inside or outside the bitstream. In the example shown, this is signaled in the sequence parameter set as shown below.

[0103] Referring again to FIG. 6, which also shows a fragment of the sequence parameter set 610. The first syntax element shown is adapive_pic_resolution_change_flag 611. When true, this flag may indicate the use of adaptive resolution, which in turn may require certain control information. In the example, such control information is conditionally present based on the value of the flag based on the if() statement in the parameter set 612 and the tile group header 601 .

[0104] When adaptive resolution is used, this example encodes the output resolution in units of samples 613. The number 613 refers to both the output_pic_width_in_luma_samples element and the output_pic_height_in_luma_samples element, which together can determine the resolution of the output image. Elsewhere in a video coding technology or standard, certain restrictions may be defined for each value. For example, the level definition may limit the number of total output samples that can be the product of the values of these two syntax elements. In addition, certain video coding technologies or standards, or external technologies or standards such as, for example, system standards, may restrict the range of a numerical value (for example, one or both measurements must be divisible by a power of 2) or aspect ratio (for example, width and the height must be in a ratio such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementations or for other reasons, and are well known in the art.

[0105] In some applications, it may be desirable for the encoder to instruct the decoder to use a certain reference picture size rather than implicitly assume that this size is the size of the output picture. In this example, the reference_pic syntax element size_present_flag 614 triggers the conditional presence of the reference picture dimensions 615 (again, the notation of the number refers to both width and height).

[0106] Finally, a table of possible width and height of the decoded image is shown. Such a table may be expressed, for example, by specifying the table (num_dec_pic_size_in_luma_samples_minus1) 616. The "minus1" element may refer to the interpretation of the value of this syntax element. For example, if the encoded value is zero, there is one entry in the table. If the value is five, there are six table entries. Then, for each "row" in the table, the width and height of the decoded image are included in syntax 617.

[0107] Represented 617 table entries can be indexed using the dec_pic_size_idx 602 syntax element in the tile group header, thereby allowing different decoded sizes—essentially scaling factors—for each tile group.

[0108] Some video coding technologies or standards, such as VP9, support spatial scalability by implementing certain forms of reference picture resampling (signaled quite differently than in the disclosed subject matter) in combination with temporal scalability to provide spatial scalability, in particular certain the reference pictures may be upsampled using ARC-style techniques to higher resolution to form the basis of the spatial enhancement layer. These upsampled images can be enhanced using conventional high resolution prediction engines to add detail.

[0109] The disclosed subject matter may be used in such an environment. In some cases, in the same or another embodiment of the invention, a value in the block header of the Network Abstraction Level (NAL, Network Abstraction Level), for example, the Temporal ID field, can be used to indicate not only the temporal, but also the spatial level. This provides certain advantages for certain system structures; for example, Selected Forwarding Units (SFU) servers built and optimized for temporal selective forwarding, selected based on the NAL Header Temporal ID value, can be used without modification for scalable environments. To make this possible, there may be a requirement to map between the encoded picture size and the temporal level indicated by the Temporal ID field in the NAL unit header.

[IT] In some video coding technologies, an access unit (AU) may refer to the encoded image(s), slice(s), tile(s), NAL unit(s), and so on, that have been captured and assembled to the corresponding bitstream of the image/slice/tile/NAL unit at the given time. This point in time may be link time.

[0111] In HEVC and some other video coding technologies, a picture sequence number (POC) value may be used to indicate a selected reference picture among a plurality of reference pictures stored in a Decoded Picture Buffer (DPB). When an Access Unit (AU) contains one or more images, slices, or tiles, each image, slice, or tile belonging to the same AU may carry the same POC value, from which it can be inferred that they were created from content of the same link time. In other words, in a scenario where two images/slices/tiles carry the same POC setpoint, this may indicate that the two images/slices/tiles belong to the same AU and have the same link time. Conversely, if two images/slices/tiles have different POC values, this may indicate that these images/slices/tiles belong to different access units (AUs) and have different link times.

[0112] In one form of implementation of the disclosed subject matter, the aforementioned rigid relationship can be relaxed, since the access unit may contain images, slices or tiles with different POC values. By allowing different POC values in the AU, it becomes possible to use the POC value to identify potentially independently decodable pictures/slices/tiles with the same presentation time. This, in turn, can support multiple scalable levels without changing reference picture selection signaling (eg, reference picture set signaling or reference picture list signaling), as described in more detail below.

[0113] However, it is still desirable to be able to identify the AU that owns an image/slice/tile with respect to other images/slices/tiles having different POC values based on the POC value alone. This can be achieved as described below.

[0114] In the same or other embodiments of the invention, an access unit number (AUC, Access Unit Count) may be signaled in a high-level syntax structure such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or delimiter. AU. The AUC value may be used to identify which NAL units, pictures, slices or tiles belong to a given AU. The AUC value may correspond to a particular link time. The AUC value may be a multiple of the ROS value. By dividing the POC value by an integer value, the AUC value can be calculated. In some cases, division operations can place a certain burden on decoder implementations. In such cases, slight restrictions on the numbering space of the AUC values may allow the division operation to be replaced by shift operations. For example, the AUC value may be equal to the Most Significant Bit (MSB) value of the POC value range.

[0115] In the same embodiment, an AU ROS cycle value (poc_cycle_au) may be signaled in a high-level syntax structure such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or AU delimiter. The poc_cycle_au element may indicate how many different and consecutive POC values may be associated with the same AU. For example, if the poc_cycle_au value is 4, images, slices, or tiles with a POC value of 0-3 inclusive are associated with an AU with an AUC value of 0, and images, slices, or tiles with a POC value of 4-7, inclusive, are associated with an AU with an AUC value of 1. Therefore, the AUC value can be inferred by dividing the POC value by the ROS_cycle_au value.

[0116] In the same or another embodiment of the invention, the value of po_cycle_au can be derived from information located, for example, in a video parameter set (VPS) that identifies the number of spatial layers or SNR levels in the encoded video sequence. Such a possible relationship is briefly described below. Although the derivation as described above may save a few bits in the VPS and therefore may increase coding efficiency, it may be advantageous to explicitly encode poc_cycle_au in an appropriate high-level syntax structure hierarchically below the video parameter set in order to be able to minimize poc_cycle_au for a given small portion of the bitmap. stream, such as images. This optimization can save more bits than can be saved by the derivation process described above because POC values (and/or POC indirectly related syntax element values) can be encoded in low level syntax structures.

[0117] In accordance with the same or another embodiment of the invention in FIG. 9 shows an example of syntax tables 900 for signaling a vps_poc_cycle_au syntax element in a VPS (or SPS set) that specifies the poc_cycle_au used for all pictures/slices in a coded video sequence, and a slice_poc_cycle_au syntax element that specifies the growth_cycle_au of the current slice in the slice header. If the POC value increases uniformly for each AU, vps_constant_poc_cycle_per_au in the VPS set is set to 1 and vps_poc_cycle_au is signaled in the VPS set. In this case slice_poc_cycle_au is not explicitly signaled and the AUC value for each AU is calculated by dividing the POC value by vps_poc_cycle_au. If the POC value does not increase uniformly with each AU, vps_constant_poc_cycle_per_au in the VPS set is set to 0. In this case, vps_access_unit_cnt is not signaled, while slice_access_unit_cnt is signaled in the slice header for each slice or picture. Each slice or image can have a different slice_access_unit_cnt value. The AUC value for each AU is calculated by dividing the POC value by slice_poc_cycle_au. In FIG. 10 is a block diagram illustrating the corresponding workflow 1000.

[0118] At step S10, the semantic analysis of the VPS/SPS set is considered and it is determined whether one or more ROS cycles per AU are constant. In step S11, it is determined whether the per-AU POC cycle is constant in the coded video sequence, and if not, in step S13, the value of the access unit number is calculated based on the poc_cycle_au_value and the POC value, and if so, in S12, the value of the level poc_cycle_au_value is calculated. sequences and values of ROS. In step S14, the VPS/SPS set is parsed and a determination is made as to whether the per-AU ROS cycle is constant, which may restart such steps as described above or proceed to alternative processing.

[0119] In the same or other embodiments of the invention, even if the POC value of an image, slice, or tile may be different, an image, slice, or tile corresponding to AUs with the same AUC value may be associated with the same decoding or output time. . Therefore, without any mutual parsing/decoding dependency between images, slices or tiles in the same AU, all or a subset of the images, slices or tiles associated with the same AU can be decoded in parallel and can be output to the same point in time.

[0120] In the same or other embodiments of the invention, even if the POC value of an image, slice, or tile may be different, an image, slice, or tile corresponding to AUs with the same AUC value may be associated with the same layout/display time. . When the link time is contained in the container format, even if the images correspond to different AUs, if the images have the same link time, the images can be displayed at the same point in time.

[0121] In the same or other embodiments of the invention, each image, slice or tile may have the same temporary identifier (temporal_id) in the same AU. All or a subset of images, slices or tiles corresponding to a point in time may be associated with the same temporal sublayer. In the same or other embodiments of the invention, each image, slice or tile may have the same or different spatial layer identifier (layer_id) in the same AU. All or a subset of the images, slices or tiles corresponding to a point in time may be associated with the same or a different spatial level.

[0122] In FIG. 8 shows an example of a video sequence structure with a combination of temporal_id, layer_id, POC, and AUC values with adaptive resolution change. In this example, an image, slice or tile in the first AU with AUC=0 may have temporal_id=0 and layer_id=0 or 1, while an image, slice or tile in the second AU with AUC=1 may have temporal_id=1 and layer_id =0 or 1, respectively. The POC value is incremented by 1 for each image regardless of the temporal_id and layer_id values. In this example, the value of poc_cycle_au may be 2. Preferably, the value of poc_cycle_au may be set to the number (spatial scalability) of levels. Therefore, in this example, the ROS value is increased by 2 and the AUC value is increased by 1.

[0123] In the above embodiments, all or a subset of the inter-frame or inter-layer prediction structure and reference picture indication may be supported by using existing Reference Picture Set (RPS) signaling in the HEVC standard or Reference Picture List (RPL) signaling. ). In an RPS or an RPL, the selected reference picture is indicated by signaling a POC number value or a POC delta (change) value between the current picture and the selected reference picture. For the disclosed subject matter, an RPS set and an RPL list may be used to indicate an inter-frame or inter-layer prediction structure without changing the signaling, but with the following limitations. If the temporal_id value of the reference picture is greater than the temporal_id value of the current picture, the current picture cannot use the reference picture for motion compensation or other predictions. If the layer_id value of the reference picture is greater than the layer_id value of the current picture, the current picture cannot use the reference picture for motion compensation or other predictions.

[0124] In the same and other embodiments, motion vector scaling based on the POC difference for temporal motion vector prediction may be disabled for multiple pictures in an access block. Therefore, although each picture may have a different POC value in the access block, the motion vector is not scaled and is not used for temporal prediction of the motion vector in the access block. This is because a reference picture with a different ROS in the same AU is considered a reference picture having the same point in time. Therefore, in an embodiment of the invention, the motion vector scaling function may return 1 when the reference picture belongs to the AU associated with the current picture.

[0125] In the same and other embodiments, motion vector scaling based on the difference of POC numbers for temporal motion vector prediction can be optionally disabled for multiple images when the spatial resolution of the reference image is different from the spatial resolution of the current image. When motion vector scaling is enabled, the motion vector is scaled based on both the difference in POC numbers and the spatial resolution ratio between the current picture and the reference picture.

[0126] In the same or another embodiment of the invention, the motion vector can be scaled based on the AUC number difference instead of the POC number difference, for temporal motion vector prediction, especially when poc_cycle_au has a non-uniform value (when vps_constant_poc_cycle_per_au==0). Otherwise (when vps_constant_cycle_per_au==1), motion vector scaling based on AUC number difference may be identical to motion vector scaling based on ROS number difference.

[0127] In the same or another embodiment of the invention, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU (with the same AUC value) with the current image is not scaled based on the AUC difference and is used for motion vector prediction without scaling or with scaling based on the ratio of spatial resolution between the current image and the reference image.

[0128] In the same and other embodiments of the invention, the AUC value is used to identify the AU boundary and is used to operate a hypothetical reference decoder (HRD) that requires input and output synchronization with AU granularity. In most cases, the decoded picture with the highest level in the AU can be output for display. The AUC value and the layer id value can be used to identify the output image.

[0129] In one form of the invention, the image may consist of one or more sub-images. Each subimage may cover a local area or an entire area of the image. The area supported by a sub-image may or may not overlap with the area supported by another sub-image. An area composed of one or more sub-images may or may not cover the entire image area. If the image consists of a sub-image, the area supported by the sub-image is identical to the area supported by the image.

[0130] In the same embodiment of the invention, the sub-image may be encoded in a coding method similar to the coding method used to encode the image. A sub-image may be independently encoded or may be encoded depending on another sub-image or encoded image. A sub-image may or may not have any parsing dependency on another sub-image or encoded image.

[0131] In the same embodiment, the encoded sub-image may be contained in one or more layers. The encoded sub-image in a layer may have a different spatial resolution. The original sub-image may be spatially resampled (upsampled or downsampled), encoded with different spatial resolutions, and contained in a bitstream corresponding to the layer.

[0132] In the same or another embodiment of the invention, a sub-image with (W, H), where W indicates the width of the sub-image and H indicates the height of the sub-image, respectively, can be encoded and contained in the encoded bitstream corresponding to level 0, while as an upsampled (or downsampled) sub-image relative to the original spatial resolution sub-image with (W*S _w,k H*S _h,k ) may be encoded and contained in the encoded bitstream corresponding to layer k, where S _{w ,k} , S _h,k indicate horizontal and vertical resampling factors. If the values of S _w,k , S _h,k are greater than 1, resampling is equivalent to upsampling. Whereas if the values of S _w,k , S _h,k are less than 1, resampling is equivalent to downsampling.

[0133] In the same or another embodiment of the invention, an encoded sub-image in a layer may have a different visual quality than the quality of an encoded sub-image in another layer in the same sub-image or another sub-image. For example, sub-image i in layer n is encoded with quantization parameter Q _i,n , while sub-image j in layer m is encoded with quantization parameter Q _j,m .

[0134] In the same or another embodiment of the invention, an encoded sub-image in a layer may be independently decodable, without any dependence on parsing or decoding from an encoded sub-image in another layer of the same local area. A sub-image layer that can be independently decodable without referring to another sub-image layer of the same local area is an independent sub-image layer. An encoded sub-image in an independent sub-image layer may or may not have a decoding or semantic analysis dependency on a previously encoded sub-image in the same sub-image layer, but an encoded sub-image may not have any dependency on an encoded image in another sub-image layer.

[0135] In the same or another embodiment of the invention, an encoded sub-image in a layer may be dependently decodable, with any parsing or decoding dependency on an encoded sub-image in another layer of the same local area. A sub-image layer that can be dependently decoded with reference to another sub-image layer of the same local area is a dependent sub-image layer. An encoded sub-image in a dependent sub-image may refer to an encoded sub-image belonging to the same sub-image, a previously encoded sub-image in the same sub-image layer, or both reference sub-images.

[0136] In the same or another embodiment of the invention, the encoded sub-image consists of one or more independent sub-image layers and one or more dependent sub-image layers. However, at least one independent sub-picture layer may be present for an encoded sub-picture. An independent sub-image layer may have a layer identifier (layer_id) value, which may be present in a NAL unit header or other high-level syntax structure, equal to 0. A sub-image layer with a layer identifier of 0 is the base sub-image layer.

[0137] In the same or another embodiment of the invention, an image may consist of one or more foreground sub-images and one background sub-image. The area supported by the background sub-image may be equal to the image area. The area supported by the foreground sub-image may overlap with the area supported by the background sub-image. The background sub-image may be a sub-image base layer, while the foreground sub-image may be a non-base (enhancement) sub-image layer. One or more non-base sub-image layers may refer to the same base layer for decoding. Each non-base subimage layer with layer_id equal to a may refer to a non-base subimage layer with layer id equal to b, where a is greater than b.

[0138] In the same or another embodiment of the invention, the image may consist of one or more foreground sub-images with or without a background sub-image. Each sub-image may have its own sub-image base layer and one or more non-base (enhancement) levels. Each base subimage level may be referenced by one or more non-base subimage levels. Each non-base subimage layer with layer_id equal to a may refer to a non-base subimage layer with layer_id equal to b, where a is greater than b.

[0139] In the same or another embodiment of the invention, the image may consist of one or more foreground sub-images with or without a background sub-image. Each encoded sub-image in a (base or non-base) sub-image layer may be referenced by one or more non-base-level sub-images that belong to the same sub-image and one or more non-base-level sub-images that do not belong to the same sub-image.

[0140] In the same or another embodiment of the invention, the image may consist of one or more foreground sub-images with or without a background sub-image. The sub-image in layer a may be further divided into multiple sub-images in the same layer. One or more encoded sub-images in layer b may refer to separated sub-images in layer a .

[0141] In the same or another embodiment of the invention, a coded video sequence (CVS) may be a group of coded pictures. The CVS may consist of one or more Coded Sub-Picture Sequence (CSPS), where the CSPS may be a group of coded sub-pictures covering the same local area of an image. The CSPS may have the same or different temporal resolution as the encoded video sequence.

[0142] In the same or another embodiment of the invention, the CSPS sequence may be encoded and may be contained in one or more layers. The CSPS sequence may consist of one or more CSPS levels. Decoding one or more CSPS levels corresponding to the CSPS may recover a sequence of sub-images corresponding to the same local area.

[0143] In the same or another embodiment of the invention, the number of CSPS levels corresponding to a CSPS may be identical or different from the number of CSPS levels corresponding to another CSPS.

[0144] In the same or another embodiment of the invention, a CSPS layer may have a different temporal resolution (eg, frame rate) than another CSPS layer. The original (uncompressed) sub-image sequence may be temporally resampled (upsampled or downsampled), encoded with different temporal resolution parameters, and contained in a layer-appropriate bitstream.

[0145] In the same or another embodiment of the invention, a sub-image sequence with a frame rate F may be encoded and contained in an encoded bitstream corresponding to level 0, while an upsampled (or downsampled) sub-image sequence of images from the original sub-image sequence, with F*S _t,k may be encoded and contained in the encoded bitstream corresponding to layer k, where S _t,k indicates the temporal sampling factor for layer k. If the value of S _t,k is greater than 1, the temporal resampling process is equivalent to frame rate upconversion. Whereas, if the value of S _t,k is less than 1, the temporal resampling process is equivalent to frame rate downconversion.

[0146] In the same or another embodiment of the invention, when the sub-image with CSPS level a is a reference (reference) to the sub-image with CSPS level b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS level a is different from the spatial resolution of CSPS level b , the decoded pixels in the CSPS layer a are resampled and used for reference. The resampling process may require upsampling or downsampling filtering.

[0147] In FIG. 11 shows an example 1100 with respect to a video bitstream including a background video CSPS sequence with layer_id equal to 0 and multiple foreground CSPS layers. While the encoded sub-picture may be composed of one or more CSPS layers, the background region that does not belong to any foreground CSPS layer may be composed of the base layer. The base layer may contain the background and foreground regions, while the CSPS enhancement layer contains the foreground region. An enhancement CSPS layer may have better visual quality than a base layer in the same area. The enhancement layer CSPS may refer to the reconstructed pixels and motion vectors of the base layer corresponding to the same area.

[0148] In the same or another embodiment of the invention, the video bitstream corresponding to the base layer is contained in a track ("track"), while the CSPS levels corresponding to each sub-image are contained in a separate track in the video file.

[0149] In the same or another embodiment of the invention, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers with the same layer_id are contained in a separate track. In this example, the track corresponding to level k includes only the CSPS levels corresponding to level k.

[0150] In the same or another embodiment of the invention, each CSPS level of each sub-image is stored in a separate track. Each track may or may not have parsing or decoding dependencies on one or more other tracks.

[0151] In the same or another embodiment of the invention, each track may contain bitstreams corresponding to levels, from level i to level j, CSPS of all or a subset of sub-images, where 0<i=<j=<k, k is the highest CSPS level .

[0152] In the same or another embodiment of the invention, an image consists of one or more associated media data, including a depth map, an alpha map, 3D geometry data, an occupancy map, and so on. Such associated synchronized media data may be divided into one or more sub-streams of data, each corresponding to one sub-picture.

[0153] In accordance with the same or another embodiment of the invention, FIG. 12 shows an example 1200 of a video conference based on a method for creating layered sub-images. The video stream contains one base layer video bit stream 1201 corresponding to the background image and one or more enhancement layer video bit streams 1202 corresponding to the foreground sub-images. Each enhancement layer video bitstream 1202 corresponds to a CSPS layer. The default display shows the image corresponding to the baseline. It contains an image of one or more users in the "picture in picture" (PIP, Picture In a Picture) mode. When a particular user is selected by the client controls, the CSPS enhancement layer 1202 corresponding to the selected user is decoded and displayed with improved quality or spatial resolution. In FIG. 13 shows a diagram 1300 of operation.

[0154] In step S20, the video bitstream is decoded using multiple layers, and in step S21, the background area and one or more foreground sub-images are identified. In step S22, it is considered whether a particular sub-image area has been selected, and if not, in step S24, the background region is decoded and displayed, and if so, in step S23, the enhanced sub-image is decoded and displayed.

[0155] In the same or another embodiment of the invention, an intermediate network unit (such as a router) may select a subset of layers to transmit to the user depending on its throughput. Image/sub-image organization can be used for bandwidth adaptation. For example, if the user does not have sufficient bandwidth, the router separates layers or selects some sub-images based on their importance or based on the settings used, and this can be done dynamically to adapt to bandwidth.

[0156] FIG. 14 shows a case of using 360-degree video 1400. When a spherical 360-degree image 1401 is projected onto a flat image, the projected 360-degree image may be divided into multiple sub-images as a base layer 1402. An enhancement layer 1403 of a particular sub-image may be encoded and transmitted to a client. . The decoder may be capable of decoding both the base layer including all sub-pictures and the enhancement layer of the selected sub-picture. When the current viewport is identical to the selected sub-image, the displayed image may have a higher quality with the decoded sub-image with an enhancement level. Otherwise, the base layer decoded image may be displayed with poor quality.

[0157] In the same or another embodiment of the invention, any layout information for display may be present in the file as additional information (eg, SEI messages or metadata). One or more decoded sub-pictures may be moved and displayed depending on the signaled layout information. The link information may be signaled by a streaming server or a broadcaster, or may be retrieved by a network entity or a cloud server, or may be determined by user customization.

[0158] In an embodiment of the invention, when an input image is divided into one or more (rectangular) sub-regions, each sub-region may be encoded as an independent layer. Each independent level corresponding to a local scope can have a unique level identifier value. For each independent layer, sub-image size and location information may be transmitted. For example, image size (width, height), information about the offset of the upper left corner (x_offset, y_offset). In FIG. 15 shows an example 1500 of arrangement of divided sub-images, information about the size and position of its sub-images, and the corresponding image prediction structure. Layout information, including sub-image size(s) and sub-image position(s), may be signaled in a high-level syntax structure such as a parameter set(s), a slice or tile group header, or an SEI message.

[0159] In the same embodiment, each sub-image corresponding to the independent layer may have its own unique AU POC value. When a reference picture among pictures stored in the DPB is specified using syntax element(s) in an RPS set or RPL structure, the POC value(s) of each sub-picture corresponding to the layer may be used.

[0160] In the same or another embodiment of the invention, to indicate the structure of the (inter-layer) prediction, the layer_id may not be used, but the (increment) value of the POC may be used.

[0161] In the same or another embodiment of the invention, a sub-image with a POC value of N corresponding to a level (or local area) may or may not be used as a reference image of a sub-image with a POC value of N+K corresponding to the same level (or the same local area), for motion-compensated prediction. In most cases, the value of the number K can be equal to the maximum number of (independent) levels, which can be identical to the number of subregions.

[0162] In accordance with the same or another embodiment of the invention, FIG. 16 shows an extended case of FIG. 15. When the input image is divided into multiple sub-regions (eg, four), each local region may be encoded with one or more levels. In this case, the number of independent levels may be equal to the number of sub-regions, and one or more levels may correspond to a sub-region. Thus, each sub-region may be encoded with one or more independent layers and zero or more dependent layers.

[0163] In the same embodiment of the invention in FIG. 16 shows an example illustration 1600, in which the input image can be divided into four sub-regions. The upper right sub-region may be encoded as two levels, which are level 1 and level 4, while the right-bottom sub-region may be encoded as two levels, which are level 3 and level 5. In this case, level 4 may refer to level 1 for motion compensated prediction, while layer 5 may refer to layer 3 for motion compensation.

[0164] In the same or another embodiment of the invention, in-loop filtering (e.g., deblocking filtering, adaptive in-loop filtering, adaptive in-loop transform (luma vs. chroma mapping), two-way filtering, or any deep learning-based filtering) with level boundary crossing can be (optional) is prohibited.

[0165] In the same or another embodiment of the invention, motion-compensated prediction or intra-block copying across a layer boundary may be (optionally) disabled.

[0166] In the same or another embodiment of the invention, boundary padding for motion-compensated prediction or in-loop filtering at the sub-image boundary can be optionally processed. A flag indicating whether boundary padding is being processed or not may be signaled in a high-level syntax structure such as a parameter set(s) (VPS, SPS, PPS or Adaptation Parameter Set (APS)), a slice or tile group header or an SEI message.

[0167] In the same or another embodiment of the invention, layout information of sub-area(s) (or sub-image(s)) may be signaled in VPS or SPS sets. In FIG. 17 shows an example of 1700 syntax elements in the VPS and SPS sets. In this example, the vps_sub_picture_dividing_flag is signaled in the VPS set. The flag may indicate whether the input images are divided into multiple subregions. When the value of vps_sub_picture_dividing_flag is 0, the input picture(s) in the encoded video sequence(s) corresponding to the current VPS cannot be divided into multiple sub-regions. In this case, the input image size may be equal to the encoded image size (image width in luma samples, image height in luma samples) that is signaled in the SPS set. When the value of vps_sub_picture_dividing_flag is 1, the input image(s) may be divided into multiple sub-regions. In this case, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled in the VPS set. The values vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples can be equal to the width and height of the input image(s), respectively.

[0168] In the same embodiment, the vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples values cannot be used for decoding, but can be used for layout and display.

[0169] In the same embodiment, when the value of vps_sub_picture_dividing_flag is equal to 1, the syntax elements pic_offset_x and pic_offset_y can be signaled in the SPS set corresponding to (a) the specific layer(s). In this case, the size of the encoded picture (pic_width_in_luma_samples, pic_height_in_luma_samples) transmitted in the SPS set may be equal to the width and height of the sub-region corresponding to the particular layer. In addition, the position (pic_offset_x, pic_offset_y) of the upper left corner of the sub-area may be signaled in the SPS set.

[0170] In the same embodiment, the position information (pic_offset_x, pic_offset_y) of the upper left corner of the sub-area cannot be used for decoding, but can be used for layout and display.

[0171] In the same or another embodiment of the invention, information about the layout (size and position) of all or a subset of the sub-area(s) of the input image(s), information about the relationship between the level(s) may be signaled in a parameter set or in SEI message. In FIG. 18 shows an example 1800 of syntax elements for specifying information about the arrangement of subareas, dependencies between levels, and relationships between a subarea and one or more levels. In this example, the num_sub_region syntax element indicates the number of (rectangular) subregions in the current encoded video sequence, the num_elements syntax element of the num_layers syntax element indicates the number of levels in the current encoded video sequence. The value of num_layers can be equal to or greater than the value of num_sub_region. When a sub-region is encoded as one level, the value of num_layers may be equal to the value of num_sub_region. When one or more subregions are encoded as multiple levels, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] specifies the j-th level dependency for the i-th level, num_layers_for_region[i] specifies the number of levels associated with the i-th sub-region; sub_region_layer_id[i][j] indicates the layer_id of the j-th layer associated with the i-th sub-region. The elements sub_region_offset_x[i] and sub_region_offset_y[i] indicate the horizontal and vertical position of the upper left corner of the i-th subregion, respectively. The elements sub_region_width[i] and sub_region_height[i] indicate the width and height of the i-th subregion, respectively.

[0172] In one embodiment of the invention, one or more syntax elements that define an output level set, set to indicate one or more levels to be output with or without profile tier level information, may be signaled in a high level syntax structure, for example, a VPS set, Dependency Parameter Set (DPS), SPS, PPS, APS, or SEI message. As shown in FIG. 19, a num_output_layer_sets syntax element indicating the number of output layer sets (OLS, Output Layer Set) in a coded video sequence referring to a VPS may be signaled in the VPS. For each set of output levels, the output_layer_flag flag can be signaled as many times as there are output levels.

[0173] In the same embodiment, output_layer_flag[i] equal to 1 indicates that the i-th level is being output. The vps_output_layer_flag[i] element equal to 0 indicates that the i-th layer is not output.

[0174] In the same or another embodiment of the invention, one or more syntax elements that define profile, tier, and level information for each set of output levels may be signaled in a high-level syntax structure, such as VPS, DPS, SPS, PPS, APS sets. or an SEI message. Still referring to FIG. 19, a num_profile_tile_level syntax element indicating a number of profile, tier, and level information for each OLS in a coded video sequence referring to a VPS may be signaled in the VPS. For each output layer set, a set of syntax elements for profile, tier, and layer information, or an index indicating specific profile, tier, and layer information among entries in the profile, tier, and layer information, may be signaled as many times as the number of output layers.

[0175] In the same embodiment, profile_tier_level_idx[i][j] specifies an index in the list of profile_tier_level() syntax structures in the VPS set, of the profile_tier_level() syntax structure that applies to the j-th level of the i-th OLS set.

[0176] In the same or different embodiment of the invention, as shown in illustration 2000 in FIG. 20, the syntax elements num_profile_tile_level and/or num_output_layer_sets may be signaled when the maximum number of levels is greater than 1 (vps_max_layers_minus1>0).

[0177] In the same or another embodiment of the invention, as shown in FIG. 20, the vps_output_layers_mode[i] syntax element indicating the output layer signaling mode for the i-th output layer set may be present in the VPS set.

[0178] In the same embodiment, vps_output_layers_mode[i] equal to 0 indicates that only the topmost layer is output with the i-th set of output levels. A vps_output_layer_mode[i] element equal to 1 indicates that all levels are output with the i-th set of output levels. The vps_output_layer_mode[i] element equal to 2 indicates that the output levels are levels with vps_output_layer flag[i][j] equal to 1, with the i-th set of output levels. Other values may be reserved.

[0179] In the same embodiment, output_layer_flag[i][j] may or may not be signaled depending on the value of vps_output_layers_mode[i] for the i-th set of output levels.

[0180] In the same or another embodiment of the invention, as shown in FIG. 20, the flag vps_ptl_signal_flag[i] may be present for the i-th set of output levels. Depending on the value of vps_ptl_signal_flag[i], the profile, tier, and level information for the i-th set of output levels may or may not be signaled.

[0181] In the same or another embodiment of the invention, as shown in illustration 2100 in FIG. 21, the number of sub-pictures, max_subpics_minus1, in the current CVS sequence may be signaled in a high-level syntax structure such as VPS, DPS, SPS, PPS, APS, or SEI message.

[0182] In the same embodiment, as shown in FIG. 21, the sub-picture ID, sub_pic_id[i], for the i-th sub-picture may be signaled when the number of sub-pictures is greater than 1 (max_subpics_minus1>0).

[0183] In the same or another embodiment of the invention, one or more syntax elements indicating a sub-image identifier belonging to each layer of each output layer set may be signaled in the VPS set. As shown in illustration 2200 in FIG. 22, sub_pic_id_layer[i][j][k] indicates the k-th sub-image present in the j-th layer of the i-th output layer set. With this information, the decoder can recognize which sub-picture can be decoded and output for each level of a particular set of output levels.

[0184] In one form of the invention, the picture header (PH, Picture Header) is a syntactic structure containing syntax elements that apply to all slices of the encoded image. Picture Packet (PU, Picture Unit) is a set of NAL units that are related to each other in accordance with a given classification rule, are sequential in decoding order, and contain exactly one encoded image. The PU packet may contain a picture header (PH) and one or more Video Coding Layer (VCL) NAL units constituting the encoded picture.

[0185] In one form of the invention, the set of SPS (sequence of the original bytes of the payload (RBSP, Raw Byte Sequence Payload)) can be available to the decoding process before it is referenced, included in at least one AU with TemporalId equal to 0 or provided by external means.

[0186] In one form of the invention, the SPS set (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with TemporalId equal to 0, in a CVS sequence that contains one or more PPS bundles related to the SPS bundle or provided through external means.

[0187] In one form of the invention, an SPS set (RBSP) may be available to the decoding process before it is referenced by one or more PPS sets, included in at least one PU with a nuh_layer_id equal to the smallest value of the nuh_layer_id blocks PPS set NALs that refer to an SPS set NAL unit in a CVS sequence that contains one or more PPSs related to the SPS set or provided by external means.

[0188] In one form of the invention, an SPS set (RBSP) may be available to the decoding process before it is referenced by one or more PPS sets included in at least one PU with TemporalId equal to 0 and nuh_layer_id equal to the smallest value of nuh_layer_id of the PPS set NAL units that belong to the SPS set NAL unit or are provided by external means.

[0189] In one form of the invention, an SPS set (RBSP) may be available to the decoding process before it is referred to as one or more PPSs included in at least one PU with TemporalId equal to 0, and nuh_layer_id equal to the smallest value of nuh_layer_id of the PPS set NAL units that refer to the SPS set NAL unit in the CVS sequence that contains one or more SPS set or externally provided PPS sets.

[0190] In the same or another embodiment of the invention, pps_seq_parameter_set_id indicates the sps_seq_parameter_set_id value for the SPS set that is being referenced. The value of pps_seq_parameter_set_id may be the same across all PPS sets referenced by encoded images in a CVS sequence.

[0191] In the same or another embodiment of the invention, all SPS NAL units with a particular sps_seq_parameter_set_id value in a CVS sequence may have the same content.

[0192] In the same or another embodiment of the invention, regardless of the num_layer_id values, the NAL units of the SPS set may share the same sps_seq_parameter_set_id value space.

[0193] In the same or another embodiment, the nuh_layer_id value of the SPS NAL units may be equal to the lowest value of the nuh_layer_id of the PPS NAL units that refer to the SPS NAL unit.

[0194] In one embodiment, when an SPS with nuh_layer_id equal to m is referenced by one or more PPS with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or a (direct or indirect) reference layer with nuh_layer_id equal to m.

[0195] In one form of the invention, the PPS set (RBSP) must be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS set NAL unit, or provided when assistance from outside sources.

[0196] In one form of the invention, the PPS set (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS set NAL unit in the CVS sequence, which contains one or more PH headers (or slice-coded NAL units) belonging to a PPS or provided by external means.

[0197] In one form of the invention, a PPS set (RBSP) may be available to the decoding process before it is referenced by one or more PH headers (or encoded slice NAL unit) included in at least one PU with nuh_layer_id equal to the smallest value of nuh_layer_id of coded slice NAL units that refer to a PPS set NAL unit in a CVS sequence containing one or more PH headers (or coded slice NAL units) belonging to the PPS set or provided by external means.

[0198] In one form of the invention, the PPS set (RBSP) may be available to the decoding process before it is referenced by one or more PH headers (or encoded slice NAL block) included in at least one PU with TemporalId equal to the TemporalId of the PPS set NAL unit, and nuh_layer_id equal to the lowest nuh_layer_id value for coded slice NAL units that refer to a PPS set NAL unit in the CVS sequence that contains one or more PH headers (or coded slice NAL units) related to the set PPS, or provided through external means.

[0199] In the same or another embodiment of the invention, ph_pic_parameter_set_id in the PH determines the value of pps_pic_parameter_set_id for the used PPS set, which is referred to. The pps_seq_parameter_set_id value may be the same across all PPS sets referenced by coded pictures in a Coded Layer Video Sequence (CLVS).

[0200] In the same or another embodiment of the invention, all PPS NAL units with a particular pps_pic_parameter_set_id value in a PU must have the same content.

[0201] In the same or another embodiment of the invention, regardless of the nuh_layer_id values, the PPS set NAL units may share the same pps_pic_parameter_set_id value space.

[0202] In the same or another embodiment, the nuh_layer_id value of a PPS NAL unit may be equal to the lowest nuh_layer_id value for coded slice NAL units that refer to a NAL unit that refers to a PPS NAL unit.

[0203] In one embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or (directly or indirectly) a layer reference with nuh_layer_id equal to m.

[0204] In one form of the invention, the PPS set (RBSP) must be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS set NAL unit, or provided when assistance from outside sources.

[0205] In one form of the invention, the PPS set (RBSP) may be available to the decoding process before it is referenced, included in at least one AU with a TemporalId equal to the TemporalId of the PPS set NAL unit in the CVS sequence, which contains one or more PH headers (or encoded slice NAL units) belonging to a PPS or provided by external means.

[0206] In one form of the invention, the PPS set (RBSP) may be available to the decoding process before it is referenced by one or more PH headers (or encoded slice NAL units) included in at least one PU with nuh_layer_id equal to the lowest value of nuh_layer_id for coded slice NAL units that refer to a PPS set NAL unit in a CVS sequence that contains one or more PHs (or coded slice NAL units) belonging to the PPS set or provided by external means.

[0207] In one embodiment, a PPS (RBSP) set may be available to a decoding process before it is referenced by one or more PHs (or coded slice NAL units) included in at least one PU with a TemporalId equal to TemporalId of a PPS NAL unit and a nuh_layer_id equal to the lowest nuh_layer_id value for coded slice NAL units that refer to a PPS NAL unit in a CVS sequence that contains one or more PH headers (or coded slice NAL units) related to the PPS set or provided with the help of external means.

[0208] In the same or another embodiment of the invention, ph_pic_parameter_set_id in the PH header defines the pps_pic_parameter_set_id value for the referenced PPS set in use. The pps_seq_parameter_set_id value may be the same across all PPS sets referenced by encoded pictures in the CLVS sequence.

[0209] In the same or another embodiment of the invention, all PPS NAL units with a particular pps_pic_parameter_set_id value in a PU must have the same content.

[0210] In the same or another embodiment of the invention, regardless of the nuh_layer_id values, PPS NAL units may share the same pps_pic_parameter_set_id value space.

[0211] In the same or another embodiment of the invention, the nuh_layer_id value of the PPS NAL unit may be equal to the lowest nuh_layer_id value for coded slice NAL units referencing the NAL unit that refers to the PPS NAL unit.

[0212] In one embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more encoded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or a (direct or indirect) reference layer for a layer with nuh_layer_id equal to m.

[0213] In one form of the invention, when the no_temporal_sublayer_switching_flag flag is signaled in DPS, VPS, or SPS sets, the TemporalId PPS value related to the parameter set containing the flag equal to 1 may be 0, while the TemporalId PPS value, related to the option set containing the flag equal to 1, may be equal to or greater than the TemporalId value of the option set.

[0214] In one form of the invention, each set of PPS (RBSP) may be available for the decoding process before it is referenced, included in at least one AU with a TemporalId less than or equal to the TemporalId of the encoded slice NAL unit ( or NAL block PH) that refers to it or is provided by external means. When a PPS NAL unit is included in an AU prior to an AU containing a coded slice NAL unit referring to the PPS, a VCL NAL unit allowing temporary switching to an upper layer, or a VCL NAL unit with nal_unit_type equal to STSA_NUT which indicates that the picture in the unit The VCL NAL may be a Step-Wise Temporal Sublayer Access (STSA) image, may not be present after the PPS NAL unit and before the coded slice NAL unit referring to the APS.

[0215] In the same or another embodiment of the invention, the PPS NAL unit and the coded slice NAL unit (and its PH NAL unit) referring to the PPS may be included in the same AU.

[0216] In the same or another embodiment of the invention, the PPS NAL unit and the STSA NAL unit may be included in the same AU that precedes the coded slice NAL unit (and its PH NAL unit) referring to the PPS.

[0217] In the same or another embodiment of the invention, the STSA NAL unit, the PPS NAL unit, and the coded slice NAL unit (and its PH NAL unit) referring to the PPS may be present in the same AU.

[0218] In the same embodiment, the TemporalId value of the VCL NAL unit containing the PPS may be equal to the TemporalId value of the previous STSA NAL unit.

[0219] In the same embodiment, the Picture Sequence Number (POC) value of the PPS NAL unit may be equal to or greater than the POC value of the STSA NAL unit.

[0220] In the same embodiment, the Picture Sequence Number (POC) value of a coded slice NAL unit or a PH NAL unit that refers to the PPS NAL unit may be equal to or greater than the POC value of the PPS NAL unit that is referenced.

[0221] APS NAL units, regardless of nuh_layer_id values, may share the same adaptation_parameter_set_id and aps_params_type value spaces.

[0222] The value of sps_video_parameter_set_id must be the same in all SPS sets referenced by encoded images in the CVS sequence at different levels.

[0223] In one embodiment, in the NAL unit header semantics of the current draft VVC specification JVET-P2001 (editively updated in JVET-Q0041), the value of nuh_layer_id for NAL units that are not VCL units is limited as follows: If nal_unit_type is equal to PPS_NUT, PREFIX_APS_NUT or SUFFIX_APS_NUT, nuh_layer_id shall be equal to the lowest value of nuh_layer_id for coded slice NAL units that refer to the NAL unit. Otherwise, if nal_unit_type is equal to SPS_NUT, nuh_layer_id shall be equal to the smallest value of nuh_layer_id of PPS NAL units that refer to the SPS NAL unit.

[0224] The constraints are intended to allow parameter sets (SPS, PPS, APS) to be referenced in different layers so that a coded slice NAL unit can/should only refer to a PPS/APS NAL unit in the same or lower layer, and a PPS NAL unit could/should only refer to an SPS set NAL unit at the same or lower layer. Note that a coded slice VCL NAL unit can refer to a PPS/APS NAL unit in a non-reference layer with given restrictions. For example, in FIG. 22 shows a simple two-layer example 2200 where layer B's nuh_layer_id value is greater than layer A's nuh_layer_id value, and layer A is not a layer B direct/indirect reference layer. In this case, the coded slice VCL NAL unit in layer B may refer to the PPS /APS with nuh_layer_id equal to level A's nuh_layer_id because there is no restriction to prevent reference to PPS/APS in the non-reference layer. Here, the nuh_layer_id PPS/APS is equal to the smallest value of the nuh_layer_id of the coded slice NAL units that belong to the NAL unit. In the example, layer A and layer B may belong to different output layer sets, and layer A NAL units may be discarded by the bitstream extraction process while the layer B NAL unit is still present in the output bitstream. Then the PPS/APS NAL units referred to by the layer B coded slice NAL unit may not be present in the output bitstream.

[0225] In the same or another embodiment of the invention, to solve the above problem, it is necessary to improve the current limitations. The proposed restriction is that a coded slice NAL unit should only refer to a PPS/APS set in the same layer or (forward) reference layer, and a PPS NAL unit should only refer to an SPS set in the same layer or (forward) reference layer. reference level. In FIG. 22, if layer A is a reference layer for layer B, both layers must belong to the same (output) layer set and must always be present in the extracted bitstream. Then the SPS/PPS/APS VCL NAL unit referenced by the NAL unit in another layer is never deleted.

[0226] According to embodiments of the invention, the semantics of the NAL unit header includes features such that when a coded slice NAL unit refers to a non-VCL NAL unit with nal_unit_type equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, the NAL unit level is not which is a VCL unit shall be equal to the coded slice NAL unit level or forward reference level for the coded slice NAL unit, and when the PPS NAL unit refers to an SPS NALy unit, the SPS NAL unit level shall be equal to the PPS NAL unit level or forward reference. level for the PPS NAL unit.

[0227] According to embodiments of the invention, the semantics of the NAL unit header includes features such that when a coded slice NAL unit refers to a NAL unit that is not a VCL unit with nal_unit_type equal to PPS_NUT, PREFIX_APS_NUT, or SUFFIX_APS_NUT, the NAL unit level is not being a VCL unit shall be equal to the coded slice NAL unit level or direct/indirect reference level for the coded slice NAL unit, and when the PPS NAL unit refers to an SPS NAL unit, the SPS NAL unit level shall be equal to the PPS NAL unit level or direct/indirect reference layer for the PPS NAL unit.

[0228] The techniques for signaling the adaptive resolution parameters described above may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, in FIG. 7 shows a computer system 700 suitable for implementing some embodiments of the disclosed subject matter.

[0229] Computer software may be coded using any suitable machine code or computer language that may be the subject of translation, compilation, linking, or similar mechanisms to produce code containing instructions that can be executed directly or through interpretation, execution of microcode, and etc. computer central processing units (CPU, Central Processing Unit), graphic processors (GPU, Graphics Processing Unit), etc.

[0230] The commands may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[0231] The components shown in FIG. 7 for computer system 700 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present invention. The component configuration is also not to be interpreted as having any dependency or requirement related to any one or combination of components shown in the exemplary computer system 700. Computer system 700 may include certain user interface input devices. Such a user interface input device may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipe, movement of an information glove), audio input (e.g., voice, hand clapping), visual input (for example: gestures), olfactory input (not shown).

[0233] User interface input devices may include one or more of (only one of each depicted): a keyboard 701, a mouse 702, a trackpad 703, a touch screen 710, an information glove 704, a joystick 705, a microphone 706, a scanner 707, cameras 708.

[0234] Computer system 700 may also include certain output devices with a user interface. Such user interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such user interface output devices may include haptic output devices (e.g., haptic feedback from a 710 touch screen, 704 data glove, or 705 joystick, but may also be haptic feedback devices that do not serve as input devices), audio output devices (such as speakers 709, headphones (not shown)), visual output devices (for example, screens 710, including cathode ray tube screens), liquid-crystal display (LCD, Liquid-Crystal Display) screens ), plasma screens, Organic Light-Emitting Diode (OLED) screens, each with or without touch capability, each with or without tactile feedback; - some of these may be capable of 2D visual output, or more than three-dimensional output by means such as stereographic output; virtual reality glasses (not pictured), holographic displays, smoke tanks (not pictured), and printers (not pictured).

[0235] The computer system 700 may also include user-accessible storage devices and associated media, such as optical media, including read-only (ROM, Read Only Memory)/read-and-write (RW, Read and Write) 720 with CDs (CD, Compact Disk)/Digital Versatile Discs (DVD, Digital Versatile Disc) or the like 721, flash drive 722, removable hard disk or solid state drive 723, traditional magnetic media such as tape and floppy disk ( not shown), read-only memory (Read Only Memory ROM)/application-specific integrated circuit (ASIC)/programmable logic device (PLD) devices, such as security keys (not shown). ) etc.

[0236] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the currently disclosed subject matter does not encompass transmission media, carrier waves, or other time signals.

[0237] The computer system 700 may also include an interface to one or more communication networks 755. Networks 755 may be, for example, wireless, wired, optical. Networks 755 may further be local, wide area, metro, automotive, industrial, real-time, latency tolerant, and so on. Examples of networks 755 include local area networks such as Ethernet, wireless local area networks (LAN, Local Area Network), cellular networks, including global system for mobile communications (GSM, Global System for Mobile communications), third-generation 3G, 3- rd Generation), fourth generation communication networks (4G, 4th Generation), fifth generation communication networks 5G, 5th Generation), long-term evolution technology communication network (LTE, Long-Term Evolution), etc., television (TV, Television) wired or wireless global digital networks, including cable television (TV), satellite television (TV) and terrestrial television, automotive and industrial, including a local area network of controllers (CANBus, Controller Area Network Bus), etc. Some networks 755 typically require external network interface adapters 754 that are connected to certain general purpose data ports or peripheral buses 749 (such as, for example, the Universal Serial Bus (USB) ports of the computer system 700; others are typically integrated into the core of the computer system). system 700 by connecting to a system bus as described below (for example, an Ethernet interface to a personal computer system (PC, Personal Computer) or a cellular network interface to a smartphone computer system.) Using any of these networks 755, computer system 700 can communicate with Such communications may be unidirectional, receive-only (eg, TV broadcast), unidirectional, transmit-only (eg, CANbus to certain CANbus devices), or bi-directional, such as with other computer systems using local area networks or wide area digital networks. protocols and stacks protocols may be used on each of these networks 755 and network interfaces 754 as described above.

[0238] The aforementioned user interface devices, user accessible storage devices, and network interfaces 754 may be attached to the core 740 of the computer system 700.

[0239] The core 740 may contain one or more central processing units (CPU, Central Processing Unit) 741, graphics processors (GPU, Graphics Processing Unit) 742, specialized programmable processing units in the form of user-programmable matrix 743 logical elements (FPGA, Field Programmable Gate Area), hardware accelerators 744 for certain tasks, etc. These devices, along with Read-Only Memory (ROM) 745, Random-Access Memory (RAM) 746, graphics adapter 750, high-capacity internal storage 747, such as internal hard drives, are not available user, solid state drives (SSD, Solid State Drive), etc., can be connected via the system bus 748. In some computer systems, the system bus 748 may be available in the form of one or more physical connectors to allow expansion with additional processors , GPUs, etc. Peripherals can be connected either directly to the kernel system bus 748 or via a peripheral bus 749. Peripheral bus architectures include an I/O bus for connecting peripherals PCI (Peripheral Component Interconnect), USB, and the like.

[0240] CPUs 741, GPUs 742, FPGAs 743, and accelerators 744 may execute certain instructions, which in combination may constitute the above machine code. This machine code may be stored in ROM 745 or RAM 746. Temporary data may also be stored in RAM 746, while permanent data may be stored in internal mass storage 747, for example. Fast storage and retrieval capability for any of the storage devices may be provided through the use of a cache memory that may be tightly coupled to one or more of the CPU 741, GPU 742, mass storage 747, ROM 745, RAM 746, and etc.

[0241] The computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specially designed and constructed for the purposes of the present invention, or they may be well known and available to those skilled in the art of computer software.

[0242] By way of example, and not limitation, a computer system having architecture 700, and in particular core 740, may provide functionality as a result of execution by processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) software embodied on one or more tangible, machine-readable media. Such computer-readable media can be media associated with user-accessible mass storage, as described above, as well as certain core storage devices 740 that are non-volatile in nature, such as internal mass storage 747 or ROM 745. various embodiments of the present invention may be stored in such devices and executed by the core 740. The computer-readable medium may include one or more memories or integrated circuits in accordance with particular needs. The software may cause core 740, and in particular the processors therein (including CPUs, GPUs, FPGAs, and the like), to execute certain processes or certain portions of specific processes described herein, including determining data structures stored in RAM 746, and changing such data structures in accordance with processes defined by the software. In addition or alternatively, a computer system may provide functionality as a result of logic embedded in, or otherwise embodied in, hardware circuitry (e.g., an accelerator 744 that may operate in place of or in conjunction with software to execute certain processes or certain parts of certain processes, described here. A reference to software may include logic and vice versa where appropriate. A reference to a computer-readable medium may include a circuit (for example, an Integrated Circuit (IC)) storing software to execute, a circuit embodying logic to execute. , or both where appropriate The present invention encompasses any suitable combination of hardware and software.

[0243] Although this disclosure describes several embodiments of the invention, there are changes, permutations and various replacement equivalents that are included in the scope of the invention. Thus, it should be appreciated that those skilled in the art will be able to develop numerous systems and methods that, although not expressly shown or described herein, embody the principles of the invention and thus fall within its spirit and scope.

Claims (27)

1. A video decoding method performed by at least one processor, the method including: obtaining encoded video data containing data of a plurality of semantically independent source images; determining, among the encoded video data, whether links are associated with any of the first access unit (AU) and the second AU according to at least one picture sequence number (ROS) signal value included in the encoded video data; and outputting a first number of links set for a first AU and a second number of links set for a second AU based on at least one POC signal value, wherein the encoded video data contains video parameter set (VPS) data identifying a plurality of spatial levels of encoded video data that are shared by the levels of adaptation parameter set (APS), picture parameter set (PPS), and sequence parameter set (SPS), wherein the same APS value space is shared by the layers, and the same sps_video_parameter_set_id value is used in SPS. 2. The method of claim 1, wherein the links contain at least one of images, slices, and tiles of encoded video data. 3. The method of claim 2, wherein determining if the links are associated with either of the first AU and the second AU comprises comparing the respective POC values of each of the links with said at least one POC signal value. 4. The method of claim 3, wherein determining whether the links are associated with either of the first AU and the second AU further comprises setting a first link count for the first AU in response to determining that each of the first link count, respectively, contains one of a plurality of POC values less than said at least one POC signal value, and setting a second link number for the second AU in response to determining that each of the second number of links, respectively, contains one of the second set of POC values equal to or exceeding said at least one ROS signal value. 5. The method according to claim 4, in which links contain slices, and said at least one POC signaling value is included in a slice header of encoded video data. 6. The method according to any one of paragraphs. 1-4, wherein at least one POC signal value is included in a Video Parameter Set (VPS) of encoded video data. 7. The method of claim 6, further comprising: determining whether the VPS data contains at least one flag indicating whether one or more links are divided into a plurality of sub-areas; and in the case where at least one flag indicates that one or more links are divided into a plurality of sub-regions, determining at least one of the full image width and the full image height, in brightness samples, of the image of one or more links. 8. The method of claim 7, further comprising: in the case where at least one flag indicates that one or more links are divided into a plurality of sub-areas, determining a signaling value included in the encoded video sequence parameter set specifying an offset of a portion of at least one of the sub-areas. 9. The method according to any one of paragraphs. 1-8, in which a plurality of semantically independent source images represent a spherical 360-degree image. 10. The method according to any one of paragraphs. 1-9, wherein the layer IDs of any of the APS, PPS, and SPS network abstraction layer (NAL) units are less than or equal to the NAL unit layer ID that refers to any of said APS, PPS, and SPS NAL units, and a layer of any of said APS, PPS and SPS NAL units is a reference layer of said NAL unit. 11. The method according to any one of paragraphs. 1-10, wherein inferring a first link number set for a first AU and a second link number set for a second AU based on at least one POC signal value is also based on whether the first link number contains one of a plurality of POC values. , which is less than the mentioned at least one signal value ROS. 12. Device for video decoding, comprising: at least one memory configured to store computer program code; at least one processor configured to access the computer program code and operate in accordance with the commands of the computer program code to implement the method according to any one of claims. 1-11. 13. A computer-readable medium that stores a program configured to cause the computer to carry out the method according to any one of paragraphs. 1-11.

Images