RU2815415C2 - Signalling for reference image oversampling with indication of constant window size in video bit stream - Google Patents

Abstract

FIELD: video encoding.

SUBSTANCE: signalling the size of an image or parts of an image that may vary from image to image. A method for decoding an encoded video bit stream is provided, which includes: obtaining the first flag indicating whether a constant image size is used in an encoded video sequence including the current image; based on the first flag indicating that a constant image size is used, decoding the current image without performing resampling of the reference image; based on the first flag indicating that a constant image size is not used, obtaining the second flag indicating whether the matching window size is signalled; based on the second flag indicating that the size of the match window is signalled, obtaining the size of the match window, determining an oversampling factor between the current image and the reference image based on the size of the match window, and performing resampling of the reference image on the current image using the oversampling factor. The first flag is signalled in the Sequence Parameter Set (SPS) and indicates whether the SPS match window parameters are signalled in the SPS.

EFFECT: improved efficiency of encoding and decoding the video bit stream.

16 cl, 10 dwg, 1 tbl

Description

[1] This application claims priority to U.S. Provisional Patent Application No. 62/903,601, filed September 20, 2019, and U.S. Patent Application No. 17/009,979, filed September 2, 2020, which are incorporated herein in their entirety.

Field of technology to which the invention relates

[1] The disclosed subject matter of the invention relates to video encoding and decoding, and more particularly to signaling the size of an image or portions of an image that may vary from image to image or from portion of an image to portion of an image.

State of the art

[2] It is known to encode and decode video using interframe prediction with motion compensation. Uncompressed digital video may consist of a series of images, each image having a spatial dimension, such as 1920 x 1080 luminance samples and associated chrominance samples. A series of images can have a fixed or variable image rate (informally also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 luminance sample resolution at 60 Hz frame rate) requires a bandwidth of close to 1.5 Gbps. An hour of such video requires more than 600 GB of disk space.

[3] One of the purposes of video encoding and decoding may be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, or a combination of both, can be used. Lossless compression refers to techniques in which an exact copy of the original signal can be reconstructed from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal suitable for the intended application. In the case of video, lossy compression is widely used. The acceptable degree of distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of broadcast television applications. The achievable compression ratio may reflect the following: Higher resolution/acceptable distortion may result in a higher compression ratio.

[4] The video encoder and video decoder may use techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding, some of which will be presented below.

[5] Historically, video encoders and video decoders have tended to operate at a given picture size, which in most cases was defined and remained constant for a coded video sequence (CVS), group of pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2 it is known that the system design changes the horizontal resolution (and thus the image size) depending on factors such as scene activity, but only for I-pictures, hence typically for GOP. Resampling of reference images to use different resolutions in CVS is known, for example, from ITU-T Rec. H.263, Annex P. However, here the image size is not changed, only the reference images are resampled, resulting in potentially only parts of the image canvas being used (in in the case of downsampling) or capturing only parts of the scene (in the case of upsampling). In addition, H.263 Annex Q allows an individual macroblock to be resampled by a factor of two (in each dimension), up or down. Again, the image size remains the same. The macroblock size is fixed in H.263 and therefore does not need to be transmitted.

[6] Image resizing in predicted images has become more common in modern video encoding. For example, VP9 allows you to resample a reference image and change the resolution of the entire image. Likewise, some proposals made regarding VVC (including, for example, Hendry, et. al., “On adaptive resolution change (ARC) for VVC,” JVT document JVET-M0135-v1, January 9-19, 2019, included herein in its entirety) allow entire reference images to be resampled to a different, higher or lower, resolution. This document proposes to encode different resolution candidates in a sequence parameter set and refer to them using syntax elements for each image in the image parameter set.

Disclosure of the invention

[7] In one embodiment, a method is provided for decoding an encoded video bitstream using at least one processor, the method including: obtaining a first flag indicating whether a constant image size is used in an encoded video sequence including the current image; based on the first flag indicating that a constant image size is used, decoding the current image without performing resampling of the reference image; based on the first flag indicating that a constant image size is not used, obtaining a second flag indicating whether the matching window size is signaled; based on the second flag indicating that the size of the match window is signaled, obtaining the size of the match window, determining a resampling ratio between the current image and the reference image based on the size of the match window, and performing resampling of the reference image on the current image using the resampling ratio.

[8] In one embodiment, an apparatus is provided for decoding an encoded video bitstream, the apparatus including at least one memory element configured to store program code; and at least one processor configured to read the program code and operate in accordance with the instructions of the program code, the program code including: a first acquisition code configured to cause the at least one processor to receive a first flag indicating whether a constant image size in the encoded video sequence including the current image; decoding code configured to cause the at least one processor, based on the first flag indicating that a constant image size is used, to decode the current image without performing resampling of the reference image; a second acquisition code configured to cause the at least one processor, based on the first flag indicating that a constant image size is not used, to obtain a second flag indicating whether the matching window size is signaled; and execution code configured to cause the at least one processor, based on the second flag indicating that the match window size is signaled, to obtain the match window size, determine a resampling ratio between the current image and the reference picture based on the match window size, and perform the resampling reference image on the current image using an oversampling factor.

[9] In one embodiment, a non-transitory computer-readable medium is provided storing instructions including: one or more instructions that, when executed by one or more processors of an apparatus for decoding an encoded video bitstream, instruct one or more processors to: obtain a first flag indicating whether a constant image size is used in the encoded video sequence including the current image; based on the first flag indicating that a constant image size is used, decode the current image without performing resampling of the reference image; based on the first flag indicating that a constant image size is not used, obtain a second flag indicating whether the matching window size is signaled; based on the second flag indicating that the match window size is signaled, obtain the match window size, determine an oversampling ratio between the current image and the reference image based on the match window size, and resample the reference image on the current image using the oversampling ratio.

Brief description of drawings

[10] Additional features, spirit and various advantages of the disclosed subject matter will be more apparent from the following detailed description and the accompanying drawings, in which:

[11] FIG. 1 is a schematic illustration of a simplified block diagram of a communication system according to an embodiment.

[12] In FIG. 2 is a schematic illustration of a simplified block diagram of a communication system in accordance with an embodiment.

[13] In FIG. 3 is a schematic illustration of a simplified block diagram of a decoder in accordance with an embodiment.

[14] In FIG. 4 is a schematic illustration of a simplified block diagram of an encoder in accordance with an embodiment.

[15] FIG. 5 is a schematic illustration of ARC/RPR parameter signaling options in accordance with an embodiment.

[16] In FIG. 6A-6B show schematic illustrations of examples of syntax tables in accordance with an embodiment.

[17] FIG. 7 is a schematic illustration of transmitting an image size and a matching window to the SPS in accordance with embodiments.

[18] FIG. 8 is a schematic illustration of transmitting an image size and a matching window to a PPS in accordance with embodiments.

[19] In FIG. 9 is a flow diagram of an exemplary process for decoding an encoded video bitstream in accordance with an embodiment.

[20] FIG. 10 is a schematic illustration of a computer system according to an embodiment.

Carrying out the invention

[21] FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. The system (100) may include at least two terminals (110 and 120) connected to each other via a network (150). For unidirectional data transmission, the first terminal (110) may encode video data at a local location for transmission to another terminal (120) via the network (150). The second terminal (120) may receive encoded video data of another terminal from the network (150), decode the encoded data, and display the reconstructed video data. Unidirectional data transfer may be common in media serving applications and the like.

[22] FIG. 1 shows a second pair of terminals (130, 140) providing support for bidirectional transmission of encoded video, which may occur, for example, during video conferencing. For bidirectional data transmission, each terminal (130, 140) may encode video data captured at a local location for transmission to another terminal via the network (150). Each terminal (130, 140) can also receive encoded video data transmitted by another terminal, can decode the encoded data, and can display the reconstructed video data on a local display device.

[23] In FIG. 1, the terminals (110-140) may be depicted as servers, personal computers, and smartphones, but the principles of the present disclosure cannot be limited thereto. Embodiments of the present disclosure find use in laptop computers, tablet computers, media players, and/or specialized videoconferencing equipment. Network (150) is any number of networks that transmit encoded video data between terminals (110-140), including, for example, wired and/or wireless communication networks. The communications network (150) may communicate over circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For purposes of this discussion, the architecture and topology of the network (150) may not be essential to the operation of the present disclosure unless explained below herein.

[24] FIG. 2 illustrates, as an example of an application for the disclosed subject matter, the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, compressed video storage on digital media including CD, DVD, memory card, etc., and so on. .

[25] The streaming system may include a capture subsystem (213), which may include a video source (201), such as a digital camera, producing, for example, a stream (202) of uncompressed video samples. This sample stream (202), represented by a thick line to emphasize the large amount of data compared to the encoded video bit streams, can be processed by an encoder (203) connected to the camera (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (204), depicted as a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server (205) for future use. One or more streaming clients (206, 208) may access the streaming server (205) to retrieve copies (207, 209) of the encoded video bitstream (204). The client (206) may include a video decoder (210) that decodes an incoming copy of the encoded video bitstream (207) and produces an outgoing stream (211) of video samples that can be displayed on a display (212) or other rendering device (not shown). In some streaming systems, the video bit streams (204, 207, 209) may be encoded according to certain video encoding/compression standards. Examples of these standards include ITU-T Rec. H.265. A video encoding standard, informally known as Universal Video Coding or VVC, is under development. The disclosed subject matter may be used in the context of VVC.

[26] FIG. 3 may be a functional block diagram of a video decoder (210) according to an embodiment of the present disclosure.

[27] The receiver (310) may receive one or more encoded video sequences to be decoded by the decoder (210); in the same or another embodiment, one encoded video sequence at a time, where the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequence may be received from a channel (312), which may be a hardware/software connection to a storage device in which the encoded video data is stored. The receiver (310) may receive the encoded video data with other data, such as encoded audio data and/or ancillary data streams, which may be sent to their respective using entities (not shown). The receiver (310) can separate the encoded video sequence from other data. To combat network jitter, a buffer memory (315) can be connected between the receiver (310) and the entropy decoder/parser (320) (“parser”). When the receiver (310) receives data from a storage/forwarding device with sufficient bandwidth and controllability or from an iso-synchronous network, the buffer memory (315) may not be needed or may be small. For use in best quality packet networks such as the Internet, a buffer (315) may be required, which may be relatively large and may preferably be of an adaptive size.

[28] The video decoder (210) may include a parser (320) for recovering symbols (321) from the entropy encoded video sequence. Categories of these symbols include information used to control the operation of the decoder (210), and potentially information to control a visualization device, such as a display (212), which is not an integral part of the decoder, but may be connected to it, as has been shown in figure 3. The control information for the rendering device(s) may be in the form of additional enhancement information (SEI messages) or portions of a video usability information (VUI) parameter set (not shown). The parser (320) may parse/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be performed in accordance with a video encoding technology or standard and may follow principles well known to those skilled in the art, including variable length encoding, Huffman encoding, arithmetic encoding with or without context sensitivity, and so on. The parser (320) may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include Groups of Pictures (GOPs), Pictures, Tiles, Slices, Bricks, Macroblocks, Coding Tree Units (CTUs), Coding Units (CUs), Blocks, Transform Units (TUs), Prediction Units (PUs), and so on. . A tile can indicate a rectangular CU/CTU area in a specific column and a number of tiles in an image. A brick may indicate a rectangular area of CU/CTU rows within a particular tile. A slice can point to one or more image bricks that are contained in a NAL unit. A sub-image can point to a rectangular area of one or more slices in an image. An entropy decoder/parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[29] The parser (320) may perform an entropy decoding/parsing operation on the video sequence received from the buffer (315) to create symbols (321).

[30] The symbol recovery 321 may include a number of different modules depending on the type of video image or portions thereof (such as: outer and inner image, outer and inner block) and other factors. Which modules are engaged and how can be controlled by subgroup control information parsed from the encoded video sequence using the parser (320). The flow of such subgroup control information between the parser (320) and the plurality of modules is not shown below for clarity.

[31] In addition to the functional blocks already mentioned, the decoder 210 can be conceptually divided into a number of functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with each other and can be at least partially integrated into each other. However, for purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into functional modules as follows.

[32] The first module is a scaling/inverting module (351). The scaling/inverse transform module (351) receives the quantized transform factor, as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, and so on, as symbol(s) (321) from the parser ( 320). It can output blocks containing sample values that can be entered into the aggregator (355).

[33] In some cases, the output samples of the scaling/demapping module (351) may be related to an intra-encoded block; that is, to a block that does not use prediction information from previously reconstructed images, but may use prediction information from previously reconstructed portions of the current image. Such prediction information may be provided by an intra-frame prediction module (352). In some cases, the intra-frame prediction module (352) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information extracted from the current (partially reconstructed) image (358). The aggregator (355), in some cases, adds, for each sample, prediction information generated by the intra-prediction module (352) to the output sample information provided by the upscaling/inverse transform module (351).

[34] In other cases, the output samples of the scaling/demapping module (351) may be related to an intercoding and potentially motion-compensated block. In such a case, the motion compensated prediction module (353) may access the reference image memory (357) to retrieve samples used for prediction. After motion compensation of the selected samples according to the symbols (321) associated with the block, these samples can be added by the aggregator (355) to the output of the scaler/inverter (here called residual samples or residual signal) to generate output information counts. The addresses in the reference picture memory from which the motion-compensated prediction module selects prediction samples may be controlled by motion vectors available to the motion-compensated prediction module in the form of symbols (321), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values retrieved from reference image memory, where precise subsample motion vectors, motion vector prediction mechanisms, and so on are used.

[35] The output samples of the aggregator (355) can be subjected to various loop filtering techniques in the loop filter module (356). Video compression technologies may include intra-loop filtering technologies that are controlled by parameters included in the encoded video bitstream and provided to the loop filtering module (356) as symbols (321) from the parser (320), but may also respond to meta-information received during decoding previous (in decoding order) parts of the encoded image or encoded video sequence, and also respond to sample values previously restored and filtered through loop filtering.

[36] The output of the loop filter module (356) may be a stream of samples, which may be output to the renderer (212) and also stored in reference picture memory for use in future inter-frame prediction.

[37] Certain encoded images, after complete reconstruction, can be used as reference images for future prediction. After the encoded picture has been completely reconstructed and the encoded picture has been identified as a reference picture (eg, by the parser (320)), the current reference picture (358) may become part of the reference picture buffer (357), and fresh current picture memory may be reallocated before starting to restore the next encoded image.

[38] Video decoder 210 may perform decoding operations according to a predetermined video compression technology, which may be documented in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by a video compression technology or standard in use, in the sense that it matches the syntax of the video compression technology or standard as specified in the video compression technology document or standard and, in particular, in its profiles document. It may also be necessary for compliance to ensure that the complexity of the encoded video sequence is within limits defined by the level of video compression technology or standard. In some cases, levels limit the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (measured, for example, in megasamples per second), the maximum reference image size, and so on. The limits set by the layers can in some cases be further limited by hypothetical reference decoder (HRD) specifications and metadata for managing the HRD buffer carried in the encoded video sequence.

[39] In an embodiment, the receiver (310) may receive additional (redundant) coded video data. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (210) to correctly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[40] FIG. 4 may be a functional block diagram of a video encoder (203) according to an embodiment of the present disclosure.

[41] The encoder (203) may receive video samples from a video source (201) (which is not part of the encoder), which may capture video image(s) for encoding by the encoder (203).

[42] The video source (201) may provide the original video sequence for encoding by the encoder (203) in the form of a digital video sample stream, which may be of any suitable bit depth (eg: 8 bits, 10 bits, 12 bits, ...), any color space (eg BT.601 Y CrCB, RGB, ...) and any suitable sample structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In a multimedia serving system, the video source (201) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (203) may be a camera that captures local image information as a video sequence. Video data may be provided as many individual images that, when viewed sequentially, convey motion. The images themselves may be organized as a spatial array of pixels, in which each pixel may contain one or more samples depending on the sample structure used, color space, etc. The relationship between pixels and samples can be readily understood by one skilled in the art. The description below is oriented towards readings.

[43] In an embodiment, the encoder (203) may encode and compress images of the original video sequence into an encoded video sequence (443) in real time or with any other timing constraints as required by the application. Providing an appropriate encoding rate is one of the functions of the controller (450). The controller controls other functional modules as described below and is functionally connected to these modules. The connection is not depicted for clarity. Parameters set by the controller may include parameters related to speed control (image skip, quantizer, lambda value of rate-distortion optimization methods, ...), image size, group of picture (GOP) layout, maximum motion vector search range, and so on . Other functions of the controller (450) can be readily recognized by one skilled in the art as they may relate to the video encoder (203) optimized for a particular system design.

[44] Some video encoders operate in what one skilled in the art would readily recognize as a "coding loop". As a simplified description, the encoding loop may consist of an encoding portion of the encoder 430 (hereinafter referred to as the "source encoder") (responsible for generating symbols based on the input image to be encoded and the reference image(s)), and a (local) decoder (433) built into an encoder (203) that reconstructs the symbols to create sample data that the (remote) decoder can also create (since any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies discussed in the subject matter disclosed) . This reconstructed stream of samples is entered into the reference image memory (434). Since decoding a symbol stream produces bit-accurate results regardless of the location of the decoder (locally or remotely), the contents of the reference picture buffer are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" using prediction during decoding. This fundamental principle of reference picture synchronicity (and the resulting drift if synchronicity cannot be maintained, for example due to channel errors) is well known to one skilled in the art.

[45] The operation of the "local" decoder (433) may be the same as that of the "remote" decoder (210), which was already described in detail above in the description of FIG. 3. Briefly referring also to FIG. 4, however, since the symbols are available, and encoding/decoding of the symbols into the encoded video sequence by the entropy encoder (445) and parser (320) can be performed losslessly, the entropy decoding portion of the decoder (210), including the channel ( 312), sink (310), buffer (315) and parser (320), cannot be fully implemented in the local decoder (433).

[46] At this point, the observation can be made that any decoding technology other than parsing/entropy decoding that is present in the decoder must also necessarily be present in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter of the invention focuses on the operation of the decoder. The description of encoding technologies may be shortened because they are the inverse of the fully described decoding technologies. Only some areas require more detailed description, which is given below.

[47] As part of its operation, the source encoder (430) may perform motion-compensated predictive encoding that predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence that have been designated “reference frames.” Thus, the encoding engine 432 encodes the differences between blocks of pixels of the input frame and blocks of pixels of the reference frame(s) that can be selected as the prediction reference(s) for the input frame.

[48] The local video decoder (433) may decode encoded frame video data, which may be referred to as reference frames, based on the symbols generated by the source encoder (430). The operations of the encoding engine (432) may preferably be lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can usually be a copy of the original video sequence with some errors. The local video decoder (433) copies the decoding processes that may be performed by the video decoder on the reference frames and may cause the reconstructed reference frames to be stored in a reference picture cache (434). Thus, the encoder (203) may locally store copies of reconstructed reference frames that have common content as reconstructed reference frames that will be received by the far-end video decoder (assuming no transmission errors).

[49] The predictor (435) may perform predictive searches for the encoding engine (432). That is, for a new image to be encoded, the predictor (435) may search the reference image memory (434) for sample data (as reference pixel block candidates) or certain metadata such as reference image motion vectors, block shapes, etc. further, which can serve as a suitable prediction standard for new images. The predictor (435) may operate on a block-by-pixel basis to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction patterns taken from a plurality of reference images stored in the reference image memory (434).

[50] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[51] The output signals of all the above functional modules may be entropy encoded in an entropy encoder (445). The entropy encoder translates the symbols generated by the various functional modules into an encoded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, and so on.

[52] The transmitter (440) may buffer the encoded video sequence(s) generated by the entropy encoder (445) to prepare it for transmission through a communication channel (460), which may be a hardware/software connection to a storage device that will store the encoded video data . The transmitter (440) may merge the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and/or ancillary data streams (sources not shown).

[53] The controller (450) may control the operation of the encoder (203). During encoding, the controller (450) may assign each encoded image a specific encoded image type, which may affect the encoding techniques that can be applied to the corresponding image. For example, images can often be classified into one of the following frame types:

[54] The internal picture (I-picture) may be one that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of internal images, including, for example, images with independent decoder update. One skilled in the art is aware of these variants of I-images and their respective applications and features.

[55] A prediction picture (P picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most one motion vector and a reference index to predict the sample values of each block.

[56] A bidirectional prediction picture (B picture) may be one that can be encoded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multi-prediction images can use more than two reference images and associated metadata to reconstruct a single block.

[57] Source images can typically be spatially divided into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. Blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the encoding assignment applied to the corresponding block images. For example, I-picture blocks may be encoded without prediction, or they may be predictively encoded with reference to already encoded blocks of the same image (spatial prediction or intra-prediction). P-picture pixel blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one previously encoded reference picture. B-picture blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one or two previously encoded reference pictures.

[58] The video encoder (203) may perform encoding operations in accordance with a predefined video encoding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (203) may perform various compression operations, including predictive encoding operations, that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore follow the syntax specified by the video encoding technology or standard being used.

[59] In an embodiment, the transmitter (440) may transmit additional data with encoded video. The video encoder (430) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant images and slices, additional enhancement information (SEI) messages, video usability information (VUI) parameter set fragments, and so on.

[60] Compressed region aggregation, or extracting multiple semantically independent parts of an image into a single video image, has received some attention recently. Particularly in the context of, for example, 360 encoding or certain surveillance applications, multiple semantically independent source images (for example, the surface of six cubes of a 360 cube-projected scene, or individual camera inputs in the case of a multi-camera surveillance setup) may require separate adaptive resolution settings to cope with different activity of each scene at a given time. In other words, coders at a given time may choose to use different resampling factors for different semantically independent images that make up the entire 360-degree scene or surveillance scene. When combined into a single image, there is a need to perform resampling of the reference image and make adaptive resolution encoding signaling available for portions of the encoded image.

[61] Below are several terms that are referenced in the remainder of this description.

[62] A sub-image may refer to, in some cases, a rectangular arrangement of samples, blocks, macroblocks, encoding units, or similar objects that are semantically grouped and that can be independently encoded at a modified resolution. One or more sub-images may form an image. One or more coded sub-pictures may form a coded picture. One or more sub-images may be assembled into an image, and one or more sub-images may be extracted from the image. In certain environments, one or more encoded sub-pictures may be collected in a compressed region without recoding to the sample level into the encoded image, and in the same or other cases, one or more encoded sub-images may be extracted from a coded image in the compressed region.

[63] In the following, reference picture resampling (RPR) or adaptive resolution resampling (ARC) may refer to mechanisms that allow the resolution of a picture or sub-picture in a coded video sequence to be changed, for example, by resampling a reference picture. In the following, the RPR/ARC parameters refer to the control information necessary to perform the adaptive resolution change, which may include, for example, filter parameters, scaling factors, output and/or reference image resolutions, various control flags, and so on.

[64] In embodiments, encoding and decoding may be performed on a single semantically independent encoded video image. Before describing the consequences of encoding/decoding multiple sub-pictures with independent RPR/ARC parameters and their implied additional complexity, options for transmitting RPR/ARC parameters must be described.

[65] In FIG. 5 shows several embodiments of RPR/ARC parameter signaling. As noted for each of the embodiments, they may have certain advantages and certain disadvantages in terms of coding efficiency, complexity and architecture. A video coding standard or technology may select one or more of these embodiments or those known in the art to convey RPR/ARC parameters. The embodiments may not be mutually exclusive and may possibly be interchangeable depending on the needs of the application, the standards used, or the choice of encoder.

[66] RPR/ARC parameter classes may include:

[67] upsampling/downsampling factors, separate or combined in X and Y dimensions

[68] upsampling/downsampling factors with added time dimension indicating upsampling/downsampling at a constant rate for a given number of images

[69] - either of the above two options may involve encoding one or more presumably short syntax elements that may indicate a table containing the coefficient(s).

[70] is the resolution in the X or Y dimension in sample units, blocks, macroblocks, coding units (CU), or any other suitable granularity of the input image, output image, reference image, coded image, in combination or individually. If there is more than one resolution (for example, one for the input image, one for the reference image), then in some cases one set of values can be inferred from another set of values. This can be conveyed, for example, by using flags. See below for a more detailed example.

[71] - "deformation" coordinates similar to those used in H.263 Annex P, again with a suitable level of detail as described above. Annex P H.263 specifies one efficient way to encode such deformation coordinates, but other, potentially more efficient ways may also be being developed. For example, reverse Huffman coding with variable length warp coordinates according to Appendix P can be replaced by binary coding of suitable length, where the length of the binary codeword can be, for example, derived from the maximum image size, perhaps multiplied by a certain factor and offset by a certain value, so that take into account “deformation” beyond the limits of the maximum image size.

[72] - Upsampling and/or downsampling filter options. In embodiments, there may be only one filter for upsampling and/or downsampling. However, in embodiments, it may be desirable to provide greater flexibility in filter design, and this may require passing filter parameters. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g. by a list of filter coefficients using suitable entropy coding techniques), the filter may be implicitly selected by appropriate upsampling/downsampling ratios that , in turn, are transmitted in accordance with any of the mechanisms mentioned above, and so on.

[73] The following description assumes encoding a finite set of upsampling/downsampling coefficients (the same coefficient that should be used in the X and Y dimensions) specified by a codeword. This codeword may be encoded with a variable length, for example using exponential Golomb code, common to certain syntax elements in video encoding specifications such as H.264 and H.265. One suitable mapping of values to upsampling/downsampling factors could, for example, correspond to Table 1:

TABLE 1 A codeword Exponential Golomb code Source/Target Resolution 0 1 1/1 1 010 1/1.5 (50% increase) 2 011 1.5/1 (50% reduction) 3 00100 1/2 (100% increase) 4 00101 2/1 (100% reduction)

[74] Many similar mappings can be designed according to the needs of the application and the capabilities of the up- and down-motion mechanisms available in the video compression technology or standard. The table can be expanded to include more values. Values can also be represented by entropy encoding mechanisms other than Exponential Golomb codes, such as using binary encoding. This may have certain advantages where resampling factors were of interest outside of the video processing engines themselves (primarily the encoder and decoder), for example through MANE. It should be noted that for situations where a change in resolution is not required, the Exponential Golomb code, which is short, can be chosen; there is only one bit in the table above. This may have an advantage in coding efficiency over using binary codes for the most common case.

[75] The number of records in the table, as well as their semantics, can be fully or partially customized. For example, the underlying table structure may be conveyed in a "high" parameter set, such as a sequence or decoder parameter set. In embodiments, one or more such tables may be defined in a video encoding technology or standard and may be selected, for example, by a decoder or sequence parameter set.

[76] The following is a description of how the upsampling/downsampling factor (ARC information) encoded as described above can be included in a video encoding technology or standard syntax. Similar considerations may apply to one or more codewords driving the upsampling/downsampling filters. See the discussion below when a filter or other data structures require relatively large amounts of data.

[77] As shown in FIG. 5, H.263 Annex P includes ARC information (502) in the form of four warp coordinates in the image header (501), specifically in the H.263 PLUSPTYPE header extension (503). This may be a reasonable design choice when a) the image header is available and b) frequent changes to the ARC information are expected. However, the overhead when using H.263 style transmission can be quite high, and the scaling factors may not apply to the edges of the image since the image header may be temporary.

[78] In the same or another embodiment, the transmission of ARC parameters may follow a detailed example, as shown in FIGS. 6A-6B. Fig. 6A-6B depict syntax diagrams in a representation type using a notation that roughly corresponds to C-style programming, such as that used in video coding standards since at least 1993. Lines in bold indicate syntax elements present in the bitstream; lines without bold often indicate control flow or setting variables.

[79] As shown in FIG. 6A, the tile group header (601), as an exemplary syntactic structure of the header applied to the (possibly rectangular) portion of the image, may conventionally include an exponential Golomb code encoded variable length syntax element dec_pic_size_idx (602 ) (in bold). The presence of this syntax element in a tile group header can be limited by the use of adaptive resolution (603) - here the flag value is not in bold, meaning that the flag is present in the bitstream at the point where it occurs in the syntax diagram. The fact whether adaptive resolution is used for this image or parts of it can be conveyed in any high-level syntactic structure inside or outside the bitstream. In the example shown, this is passed in a set of sequence parameters as shown below.

[80] FIG. 6B also shows a sample of a sequence parameter set (610). The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, this flag may indicate the use of adaptive resolution, which in turn may require certain control information. In the example, such control information is present conditionally based on the value of a flag based on the if() statement in the parameter set (612) and the tile group header (601).

[81] When adaptive resolution is used, this example encodes the output resolution in sample units (613). Item number 613 refers to both output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can determine the resolution of the output image. Elsewhere in the video encoding technology or standard, certain limits may be specified for any value. For example, the level definition may limit the number of total output samples that can be the product of the values of these two syntax elements. Also, certain video encoding technologies or standards, or external technologies or standards, such as system standards, may limit the numbering range (for example, one or both dimensions must be divisible by a power of 2) or the width-to-height ratio (for example, width and height should be in a ratio such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementations or for other reasons, and are well known in the art.

[82] In some applications, it may be desirable for the encoder to instruct the decoder to use a certain reference image size, rather than implicitly assuming that this size is the output image size. In this example, the syntax element reference_pic_size_present_flag (614) conveys the conditional presence of the dimensions (615) of the reference picture (again, the position number refers to both width and height).

[83] Finally, a table of possible decoding image widths and heights is shown. Such a table can be expressed, for example, by specifying the table (num_dec_pic_size_in_luma_samples_minus1) (616). "Minus1" may refer to the interpretation of the meaning of this syntax element. For example, if the encoded value is zero, there is one entry in the table. If the value is five, there are six table entries. Then, for each "row" in the table, the width and height of the decoded image are included in the syntax (617).

[84] Presented table entries (617) may be indexed using the dec_pic_size_idx syntax element (602) in the tile group header, thereby allowing different decoded sizes—essentially scaling factors—for each tile group.

[85] Some video encoding technologies or standards, such as VP9, support spatial scalability by implementing certain forms of reference image resampling (transmitted entirely differently than in the disclosed subject matter) in combination with temporal scalability to provide spatial scalability. In particular, some reference images may be upsampled using ARC-style techniques to a higher resolution to form the base of the spatial enhancement layer. These upsampled images can be enhanced by using normal high-resolution prediction engines to add detail.

[86] The embodiments discussed herein may be used in such an environment. In some cases, in the same or another embodiment, a value in the NAL unit header, such as a temporal identifier field, may be used to indicate not only a temporal layer, but also a spatial layer. This may have certain benefits for certain system configurations; for example, an existing Selected Forwarding Units (SFU) architecture designed and optimized for selected temporal layers forwarded based on the NAL unit header temporary identifier value can be used without modification for scale-out environments. To make this possible, there may be a requirement for a mapping between the encoded image size and the temporal layer indicated by the temporary identifier field in the NAL unit header.

[87] Compressed region aggregation, or extracting multiple semantically independent parts of an image into a single video image, has received some attention recently. Particularly in the context of, for example, 360 encoding or certain surveillance applications, multiple semantically independent source images (for example, the surface of six cubes of a 360 cube-projected scene, or individual camera inputs in the case of a multi-camera surveillance setup) may require separate adaptive resolution settings to cope with different activity of each scene at a given time. In other words, coders at a given time may choose to use different resampling factors for different semantically independent images that make up the entire 360-degree scene or surveillance scene. When combined into a single image, there is a need to perform resampling of the reference image and make adaptive resolution encoding signaling available for portions of the encoded image.

[88] In embodiments, not all samples of the reconstructed image are intended for output. The encoder can specify the rectangular portion of the image to be output using a match window. The correspondence window may be described or indicated, for example, by an offset to the left and right of the edges of the image, as determined by the size of the image. Certain use cases can be identified where correspondence windows may be relevant, including stretched sweeps, spatial assembly of views in Multiview systems, or 360 systems where correspondence windows may point to one of several cubemap surfaces for output.

[89] Since not all applications require the use of a match window, and since match window parameters may require a certain number of bits in the bitstream and therefore, when not used, it may harm encoding efficiency, and the presence of such parameters may be limited by a flag.

[90] In embodiments, the size of the match window may be signaled in a picture parameter set (PPS). Match window parameters, which can specify the size of the match window, can be used to calculate the oversampling factor when the match window size of the reference image is different from the window size of the current image. The decoder may need to recognize the size of the matching window of each image to determine whether a resampling process is required. When the oversampling factor is not 1, the size of the output image is not constant in the encoded video sequence (CVS), and special processing and post-processing of the output images, such as up/down scaling for display, may be used.

[91] In embodiments, a flag that indicates whether the decoded/output image has the same size and resampling factor is 1 in the coded video sequence (CVS)/bitstream may be signaled in a high-level parameter set, such as a decoding parameter set ( DPS), Video Parameter Set (VPS), or Sequence Parameter Set (SPS). The flag can be used for session negotiation for video streaming or decoder configuration and display settings.

[92] Referring to FIG. 7, constant_pic_size_flag (704) equal to 1 can indicate that the image sizes in CVS are the same. That being said, a constant_pic_size_flag of 0 may indicate that the image sizes for images in CVS may or may not be the same. If the value of constant_pic_size_flag is 1, the sps_conformance_window_flag (705) may be present in SPS (701). In this case, sps_conformance_window_flag equal to 1 may indicate that the conformance crop window offset parameters follow in a suitable position, for example, further, in SPS. However, an sps_conformance_window_flag of 0 may indicate that there are no conformance crop window offset parameters.

[93] In embodiments, sps_conf_win_left_offset (706), sps_conf_win_right_offset (707), sps_conf_win_top_offset (708), and sps_conf_win_bottom_offset (709) may indicate image samples in CVS that are output from the image decoding process as a rectangular area specified in the image coordinates for output .

[94] In embodiments, when the syntax elements sps_conf_win_left_offset, sps_conf_win_right_offset, sps_conf_win_top_offset, and sps_conf_win_bottom_offset are missing, the values of sps_conf_win_left_offset, sps_conf_win_right_offset, sps_conf_win_top_offset, and sps_conf_win_bottom_offset may be output equal to 0.

[95] In embodiments as shown in FIG. 8, pic_width_in_luma_samples (802) may determine the width of each decoded image based on the PPS (801) in units of luma samples. In embodiments, pic_width_in_luma_samples may not be equal to 0, may be an integer multiple of Max(8, MinCbSizeY), and may be less than or equal to pic_width_max_in_luma_samples. If missing, pic_width_in_luma_samples may be output as equal to pic_width_max_in_luma_samples. In this case, pic_height_in_luma_samples (803) may indicate the height of each decoded PPS-based image in units of luminance samples. In this case, pic_height_in_luma_samples in some cases may not be equal to 0, but may be an integer multiple of Max (8, MinCbSizeY), and may be less than or equal to pic_height_max_in_luma_samples. If missing, the value of pic_height_in_luma_samples may be output as equal to pic_height_max_in_luma_samples.

[96] In embodiments, as still shown in FIG. 8, a conformance_window_flag (804) equal to 1 may indicate that the conformance trim window offset parameters follow in an appropriate location, such as following in the PPS (801). However, a conformance_window_flag of 0 may indicate that there are no conformance crop window offset parameters. However, conf_win_left_offset (805), conf_win_right_offset (806), conf_win_top_offset (807) and conf_win_bottom_offset (808) can point to PPS-based image samples that are output from the decoding process, as a rectangular area specified in the output image coordinates.

[97] In the same embodiment, when the syntax elements conf_win_left_offset, conf_win_right_offset, conf_win_top_offset and conf_win_bottom_offset are missing, the values of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset and conf_win_bottom_offset can be output equal to the values of sps_conf_win_left_offset, sps _conf_win_right_offset, sps_conf_win_top_offset and sps_conf_win_bottom_offset respectively.

[98] In embodiments, the match trim window may contain luminance samples with horizontal image coordinates from SubWidthC * conf_win_left_offset to pic_width_in_luma_samples - (SubWidthC * conf_win_right_offset + 1) and vertical image coordinates from SubHeightC * conf_win_top_offset to pic_height_in_luma_samples - ( SubHeightC * conf_win_bottom_offset + 1 ) , inclusive.

[99] The value of SubWidthC * (conf_win_left_offset + conf_win_right_offset) can be less than pic_width_in_luma_samples, and the value of SubHeightC * (conf_win_top_offset + conf_win_bottom_offset) can be less than pic_height_in_luma_samples.

[100] The PicOutputWidthL and PicOutputHeightL variables can be obtained as shown in Equation 1 and Equation 2 below:

PicOutputWidthL = pic_width_in_luma_samples - SubWidthC * (conf_win_right_offset + conf_win_left_offset) (Equation 1)

PicOutputHeightL = pic_height_in_luma_samples - SubHeightC * (conf_win_bottom_offset + conf_win_top_offset) (Equation 2)

[101] In embodiments, the fractional interpolation process with resampling of the reference image may be processed as follows.

[102] The input to this process may be a luminance location (xSb, ySb) indicating the top left luminance reference of the current encoding subblock relative to the top left luminance reference of the current image, a variable sbWidth indicating the width of the current encoding subblock, a variable sbHeight indicating the height of the current encoding subblock , the motion vector offset mvOffset, the refined motion vector refMvLX, the selected reference image sample array refPicLX, the half-sample interpolation filter index hpelIfIdx, the bidirectional optical flow flag bdofFlag, and the variable cIdx indicating the color component index of the current block.

[103] The output of this process can be: an array of predSamplesLX (sbWidth + brdExtSize) x (sbHeight + brdExtSize) prediction sample values.

[104] The prediction block boundary extension size brdExtSize can be obtained as shown in Equation 3 below:

brdExtSize = ( bdofFlag | | ( inter_affine_flag[ xSb ][ ySb ] && sps_affine_prof_enabled_flag ) ) ? 2:0 (Equation 3)

[105] The variable fRefWidth can be set equal to the PicOutputWidthL of the reference image in luminance samples. The variable fRefHeight can be set to the PicOutputHeightL of the reference image in luminance samples. The motion vector mvLX can be set to (refMvLX - mvOffset).

[106] If cIdx is 0, the following may apply:

- Scaling factors and their fixed-point representations can be determined according to Equation 4 and Equation 5 below:

hori_scale_fp = ( ( fRefWidth << 14 ) + ( PicOutputWidthL

>> 1 ) ) / PicOutputWidthL (Equation 4)

vert_scale_fp = ( ( fRefHeight << 14 ) + ( PicOutputHeightL

>> 1 ) ) / PicOutputHeightL (Equation 5)

- Let ( xIntL, yIntL ) be the luminance location, given in integer count units, and ( xFracL, yFracL ) could be the offset, given in 1/16 count units. These variables can be used at this point to specify locations of fractional samples within refPicLX arrays of reference samples.

- The top left coordinate of the bounding box to fill with the reference sample (xSbInt _L ,ySbInt _L ) can be set to ( xSb + ( mvLX[ 0 ] >> 4 ), ySb + ( mvLX[ 1 ] >> 4 ) ).

- For each luminance sample location (xL ₌ 0..sbWidth - 1 + brdExtSize, yL ₌ 0..sbHeight - 1 + brdExtSize) within the predicted luminance sample array predSamplesLX, the corresponding predicted luminance sample value predSamplesLX [ _xL ] [ _yL ] is obtained as follows :

- Let (refxSb _L , refySb _L ) and (refx _L ,refy _L ) be the luminance locations pointed to by the motion vector (refMvLX, refMvLX) specified in 1/16 count units. The variables refxSb _L , refx _L , refySb _L and refy _L can be obtained as shown in equations 6-9 below:

refxSb _L = ( ( xSb<< 4 ) + refMvLX[ 0 ] ) * hori_scale_fp (Equation 6)

refx _L = ( ( Sign( refxSb ) * ( ( Abs( refxSb ) + 128 ) >> 8 )

+ x _L * ( ( hori_scale_fp + 8 ) >> 4 ) ) + 32 ) >> 6 (Equation 7)

refySb _L = ( ( ySb << 4 ) + refMvLX[ 1 ] ) * vert_scale_fp (Equation 8)

refyL = ( ( Sign( refySb ) * ( ( Abs( refySb ) + 128 ) >> 8 ) + yL *

( (vert_scale_fp + 8 ) >> 4 ) ) + 32 ) >> 6 (Equation 9)

- The variables xInt _L , yInt _L , xFrac _L and yFrac _L can be obtained as shown in equations 10-13 below:

xInt _L = refx _L >> 4 (Equation 10)

yInt _L = refy _L >> 4 (Equation 11)

xFrac _L = refx _L & 15 (Equation 12)

yFrac _L = refy _L & 15 (Equation 13)

- If bdofFlag is TRUE or (sps_affine_prof_enabled_flag is TRUE and inter_affine_flag [xSb] [ySb] is TRUE), and one or more of the following conditions are true, the value of the predicted luminance sample predSamplesLX [x _L ] [y _L ] can be obtained by accessing the process of extracting an integer luminance sample as specified in the appropriate clause of the video encoding specification, with (xInt _L + ( xFrac _L >> 3) - 1),yInt _L + (yFrac _L >> 3) - 1) and refPicLX as input .

1. x _L equals 0.

2. x _L equals sbWidth + 1.

3. y _L equals 0.

4. y _L is equal to sbHeight + 1.

- Otherwise, the value of the predicted luminance sample predSamplesLX [xL] [yL] can be obtained by referring to the 8-position luminance sample interpolation filtering process as specified in the corresponding clause of the video encoding specification with (xIntL - (brdExtSize > 0 ? 1 : 0), yIntL - (brdExtSize > 0 ? 1:0)), (xFracL, yFracL), (xSbInt _L , ySbInt _L ), refPicLX, hpelIfIdx, sbWidth, sbHeight and (xSb, ySb) as inputs.

[107] Otherwise (cIdx is not 0), the following may apply:

1. Let (xIntC, yIntC) be the chromaticity location, specified in integer count units, and (xFracC, yFracC) be the offset, specified in 1/32 count units. These variables can be used at this point to indicate general locations of fractional samples within refPicLX arrays of reference samples.

2. The upper left coordinate of the bounding box to fill with the reference count (xSbIntC, ySbIntC) is set to ((xSb / SubWidthC) + (mvLX [0] >> 5), (ySb / SubHeightC) + (mvLX [1]) >> 5 )).

3. For each chroma sample location (xC = 0..sbWidth - 1, yC = 0 .. sbHeight - 1) within the predicted chroma sample arrays predSamplesLX, the corresponding predicted chroma sample value predSamplesLX [xC] [yC] can be obtained as follows:

- Let (refxSb _C , refySb _C ) and (refx _C ,refy _C ) be the chroma locations pointed to by the motion vector (mvLX[0], mvLX[1]) specified in 1/32 count units. The variables refxSb _C , refySb _C , refx _C and refy _C can be obtained as shown in equations 14-17 below:

refxSb _C =( (xSb / SubWidthC << 5) + mvLX [0]) * hori_scale_fp (Equation 14)

refx _C = ((Sign (refxSb _C )* ((Abs (refxSb _C ) + 256) >> 9)

+ xC * ((hori_scale_fp + 8) >> 4)) + 16) >> 5 (Equation 15)

refySb _C =( (ySb / SubHeightC<< 5) + mvLX [1]) * vert_scale_fp (Equation 16)

refy _C = ((Sign (refySb _C )* ((Abs (refySb _C ) + 256) >> 9)

+ yC * ((vert_scale_fp + 8) >> 4)) + 16) >> 5 (Equation 17)

The variables xInt _C , yInt _C , xFrac _C and yFrac _C can be obtained as shown in Equations 18-21 below:

xInt _C = refx _C >> 5 (Equation 18)

yInt _C = refy _C >> 5 (Equation 19)

xFrac _C = refy _C & 31 (Equation 20)

yFrac _C = refy _C & 31 (Equation 21)

[108] The predicted sample value predSamplesLX [xC] [yC] can be obtained by calling the above process with (xIntC, yIntC), (xFracC, yFracC), (xSbIntC, ySbIntC), sbWidth, sbHeight and refPicLX as inputs.

[109] In FIG. 9 shows an exemplary flow diagram of a process 900 for decoding an encoded video bitstream. In some implementations, one or more process blocks of FIG. 9 may be executed by decoder 210. In some implementations, one or more process blocks of FIG. 9 may be performed by another device or group of devices separate from or including decoder 210, such as encoder 203.

[110] As shown in FIG. 9, process 900 may include obtaining a first flag indicating whether a constant image size is used in the encoded video sequence including the current image (step 910).

[111] As further shown in FIG. 9, process 900 may include determining from the first flag whether a constant image size is being used (step 920).

[112] As further shown in FIG. 9, process 900 may include, based on a first flag indicating that a constant picture size is being used (YES at step 920), decoding the current picture without resampling the reference picture (step 930).

[113] As further shown in FIG. 9, based on the first flag indicating that a constant image size is not used (NO at block 920), process 900 may proceed to block 940, block 950, block 960, and block 970.

[114] As further shown in FIG. 9, process 900 may include obtaining a second flag indicating whether the match window size is being signaled (block 940).

[115] As further shown in FIG. 9, process 900 may include, based on a second flag indicating that the match window size is signaled, obtaining the match window size (block 950), determining an oversampling ratio between the current image and the reference picture based on the match window size (block 960), and resampling the reference image for the current image using the specified resampling factor (block 970).

[116] In one embodiment, the size of the matching window may be signaled as at least one offset distance from the boundary of the current image.

[117] In one embodiment, the first flag may be signaled in a sequence parameter set (SPS), and the second flag may be signaled in one of an SPS and a picture parameter set (PPS).

[118] In one embodiment, the second flag may be signaled to the SPS and may indicate whether the SPS match window parameters are signaled to the SPS.

[119] In one embodiment, based on the second flag indicating that the SPS match window parameters are signaled to the SPS, the match window size may be obtained based on the SPS match window parameters.

[120] In one embodiment, based on the first flag indicating that the image size is not constant, process 900 may include obtaining a third flag indicating whether PPS match window parameters are signaled in the PPS.

[121] In one embodiment, based on the second flag indicating that the SPS match window parameters are signaled to the SPS and the third flag indicating that the PPS match window parameters are not signaled to the PPS, the match window size may be obtained based on the parameters SPS compliance windows.

[122] In one embodiment, based on the second flag indicating that the SPS match window parameters are not signaled to the SPS and the third flag indicating that the PPS match window parameters are signaled to the PPS, the match window size may be obtained based on the parameters PPS compliance windows.

[123] Although FIG. 9 shows exemplary blocks of process 900, in some implementations process 900 may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those depicted in FIG. 9. Additionally or alternatively, two or more blocks of process 900 may be executed in parallel.

[124] In addition, the proposed methods may be implemented by processing circuits (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program that is stored on a non-transitory computer-readable medium to implement one or more of the proposed methods.

[125] The methods described above may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 10 illustrates a computer system 1000 suitable for implementing certain embodiments of the disclosed subject matter.

[126] Computer software may be encoded using any suitable machine code or computer language that may be subject to assembly, compilation, linking or similar mechanisms to create code containing instructions that can be executed directly or through interpretation, execution of microcode, and etc., computer central processing units (CPU), graphic processing units (GPU), etc.

[127] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[128] The components shown in FIG. 10 for computer system 1000 are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of components should also not be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the exemplary embodiment of computer system 1000.

[129] The computer system 1000 may include certain human-machine interface input devices. Such a human-machine interface input device may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, control glove movement), audio input (e.g., voice, hand clapping), visual input (for example: gestures), olfactory input (not shown). Human-machine interface devices can also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, ambient sound), images (e.g. scanned images, photographic images captured from a camera for still images), video (e.g. 2D video, 3D video, including stereoscopic video).

[130] Human interface input devices may include one or more of (only one of each): keyboard 1001, mouse 1002, trackpad 1003, touch screen 1010 and associated graphics adapter 1050, control glove, joystick 1005, microphone 1006 , scanner 1007, camera 1008.

[131] The computer system 1000 may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such HMI output devices may include haptic output devices (e.g., haptic feedback from a touch screen (1010), control glove (1004), or joystick (1005), but there may also be haptic feedback devices that do not serve as input devices), audio output devices (for example: speakers (1009), headphones (not shown)), image output devices (for example, screens (1010), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without haptic feedback capability - some of which may be capable of providing two-dimensional visual output or more than three-dimensional output through means such as stereographic output; virtual reality glasses (not pictured), holographic displays and smoke tanks (not pictured)) and printers (not pictured).

[132] The computer system (1000) may also include human accessible storage devices and associated media, such as optical media, including CD/DVD ROM/RW (1020) with CD/DVD or similar media (1021), flash drive (1022), removable hard drive or solid state drive (1023), previously developed magnetic media such as tape and floppy disk (not shown), specialized ROM/ASIC/PLD based devices such as dongles (not shown) and so on.

[133] Those skilled in the art will also understand that the term "machine readable medium" as used in connection with the subject matter disclosed herein does not cover transmission media, carrier waves, or other time signals.

[134] Computer system 1000 may also include interface(s) to one or more communications networks (1155). Networks can be, for example, wireless, wired, optical. Additionally, networks can be LAN, WAN, metropolitan, automotive and industrial, real-time, latency tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless local area networks (LAN), cellular networks, including global systems for mobile communications (GSM), third generation (3G), fourth generation (4G), fifth generation (5G), Long Term Evolution (LTE) networks, etc., wired television or wireless global digital networks, including cable television, satellite television and terrestrial broadcast television, automotive and industrial, including CAN bus, and so on. Some networks typically require external network interface adapters (1154) that are connected to certain general purpose data ports or peripheral buses (1149) (for example, to the universal serial bus (USB) ports of the computer system 1000); others are typically integrated into the core of the computer system 1000 by connecting to a system bus, as described below (for example, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). For example, network 1055 may be connected to peripheral bus 1049 using network interface 1054. Using any of these networks, computer system 1000 can communicate with other entities. Such communication may be unidirectional, receive only (for example, broadcast television), unidirectional, send only (for example, CAN bus to certain CAN bus devices), or bidirectional, such as for other computer systems using a local or wide area digital network . Each of these networks and network interfaces (1154) may use specific protocols and protocol stacks, as described above.

[135] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core 1040 of the computer system 1000.

[136] The core (1040) may include one or more central processing units (CPUs) (1041), graphics processing units (GPUs) (1042), specialized field programmable gate array (FPGA) processors (1043), hardware accelerators (1044) for specific tasks, etc. These devices, along with read only memory (ROM) 1045, random access memory (RAM) 1046, internal mass storage such as internal hard drives not accessible to the user, solid state drives (SSD ) etc. 1047 may be connected via the system bus 1048. On some computer systems, the system bus 1048 may be available as one or more physical connectors to allow expansion with additional CPUs, GPUs, etc. Peripherals may be connected either directly to the core system bus 1048 or via a peripheral bus 1049. The peripheral bus architecture includes peripheral component interconnection (PCI), USB, and the like.

[137] The CPU 1041, GPU 1042, FPGA 1043, and accelerators 1044 may execute certain instructions that, in combination, may constitute the aforementioned computer code. This computer code may be stored in ROM 1045 or RAM 1046. Transient data may also be stored in RAM 1046, while persistent data may be stored, for example, in internal mass storage 1047. Rapid storage and retrieval to either storage device may be accessed by using a cache memory that may be closely coupled to one or more of the CPU 1041, GPU 1042, mass storage device 1047, ROM 1045, RAM 1046, etc.

[138] A computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be well known and available to those skilled in the computer software art.

[139] By way of example, and not limitation, a computer system having architecture 1000 and, in particular, core 1040 may provide functionality as a result of having processor(s) (including CPU, GPU, FPGA, accelerators, etc.) executing software embodied on one or more tangible, machine-readable media. Such computer-readable media may be media associated with user-accessible mass storage, as described above, as well as certain core storage devices 1040 that are non-transitory in nature, such as internal mass storage 1047 or ROM 1045. Software implementing various embodiments implementation of the present disclosure may be stored in such devices and executed by core 1040. Computer readable media may include one or more storage devices or chips according to specific needs. Software may cause the core 1040 and, in particular, processors therein (including CPUs, GPUs, FPGAs, etc.) to execute certain processes or certain portions of specific processes described herein, including defining data structures stored in RAM 1046 and modifying such data structures according to processes defined by the software. In addition or alternatively, a computer system may provide functionality as a result of logic built into hardware circuitry or otherwise embodied in circuitry (e.g., accelerator 1044), which may operate instead of or in conjunction with software to perform certain processes or portions of specific processes described in this document. A reference to software may include logic and vice versa where applicable. Reference to a computer-readable medium may include circuitry (eg, an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both, where applicable. The present disclosure includes any suitable combination of hardware and software.

[140] Although this disclosure describes several exemplary embodiments, there are variations, permutations, and various substitute equivalents that fall within the scope of the disclosure. Thus, it will be appreciated that those skilled in the art will be able to develop numerous systems and methods that, although not expressly shown or described herein, embody the principles of the disclosure and are thus within its spirit and scope of protection.

Claims (35)

1. A method for decoding an encoded video bitstream using at least one processor, the method comprising: based on determining that a constant image size is not used in the encoded video sequence including the current image, obtaining a first flag indicating whether the matching window size is signaled; based on the first flag indicating that the matching window size is being signaled: getting the matching window size, determining an oversampling ratio between the current image and the reference image based on the size of the correspondence window, and performing resampling of the reference image on the current image using a resampling factor; wherein the first flag is signaled in the sequence parameter set (SPS) and indicates whether the SPS match window parameters are signaled in the SPS. 2. The method of claim 1, wherein the size of the matching window is signaled as at least one offset distance from the boundary of the current image. 3. The method of claim 1, wherein based on the first flag indicating that the SPS match window parameters are signaled to the SPS, the match window size is obtained based on the SPS match window parameters. 4. The method of claim 1, further comprising, based on the determination that a constant image size is not used, obtaining a second flag indicating whether PPS match window parameters are signaled in the picture parameter set (PPS). 5. The method of claim 4, wherein based on the first flag indicating that the SPS match window parameters are signaled to the SPS and the second flag indicating that the PPS match window parameters are not signaled to the PPS, the match window size is obtained based on the parameters SPS compliance windows. 6. The method of claim 4, wherein based on the first flag indicating that the SPS match window parameters are not signaled to the SPS and the second flag indicating that the PPS match window parameters are signaled to the PPS, the match window size is obtained based on the window parameters PPS compliance. 7. A device for decoding an encoded video bitstream, the device comprising: at least one memory element configured to store program code; And at least one processor configured to read the program code and operate in accordance with the instructions of the program code, the program code including: a first acquisition code configured to cause the at least one processor, based on determining that a constant image size is not used in the encoded video sequence including the current image, to obtain a first flag indicating whether the matching window size is signaled; And execution code configured to cause the at least one processor, based on the first flag indicating that the size of the match window is signaled, to obtain the size of the match window, determine a resampling factor between the current image and the reference picture based on the size of the match window, and resample the reference images on the current image using the resampling factor; wherein the first flag is signaled in the sequence parameter set (SPS) and indicates whether the SPS match window parameters are signaled in the SPS. 8. The apparatus of claim 7, wherein the size of the matching window is signaled as at least one offset distance from the boundary of the current image. 9. The apparatus of claim 7, wherein based on the first flag indicating that the SPS match window parameters are signaled to the SPS, the match window size is obtained based on the SPS match window parameters. 10. The apparatus of claim 7, wherein the program code further includes a second acquisition code configured to cause the at least one processor, based on a determination that a constant image size is not being used, to obtain a second flag indicating whether the in the Picture Parameter Set (PPS) PPS match window parameters. 11. The apparatus of claim 10, wherein based on the first flag indicating that the SPS match window parameters are signaled to the SPS and the second flag indicating that the PPS match window parameters are not signaled to the PPS, the match window size is obtained based on the parameters SPS compliance windows. 12. The apparatus of claim 10, wherein based on the first flag indicating that the SPS match window parameters are not signaled to the SPS and the second flag indicating that the PPS match window parameters are signaled to the PPS, the match window size is obtained based on the window parameters PPS compliance. 13. A method for encoding video using at least one processor, the method comprising: based on determining that a constant image size is not used, signaling a matching window size; encoding the current image to generate an encoded video sequence, the encoded video sequence including the current image and a first flag indicating whether a matching window size is signaled; wherein the first flag is signaled in the sequence parameter set (SPS) and indicates whether the SPS match window parameters are signaled in the SPS. 14. A video encoding device, the device comprising: at least one memory element configured to store program code; And at least one processor configured to read the program code and operate in accordance with the instructions of the program code, the program code including: signaling code configured to cause the at least one processor, based on a determination that a constant image size is not used, to signal a matching window size; encoding code configured to cause the at least one processor to encode the current image to generate an encoded video sequence, the encoded video sequence including the current image and a first flag indicating whether a match window size is signaled; wherein the first flag is signaled in the sequence parameter set (SPS) and indicates whether the SPS match window parameters are signaled in the SPS. 15. A non-transitory computer-readable medium storing instructions, the instructions comprising: one or more instructions that, when executed by one or more processors of the apparatus for decoding an encoded video bitstream, cause the one or more processors to perform the method of any one of claims 1 to 6. 16. A non-transitory computer-readable medium storing instructions, the instructions comprising: one or more instructions that, when executed by one or more processors of the apparatus for decoding an encoded video bitstream, cause the one or more processors to perform the method of claim 13.