RU2793903C1 - Method for padding processing by parts of sub-areas in a video stream - Google Patents

Abstract

FIELD: video coding.

SUBSTANCE: encoded syntax elements corresponding to the cyclic padding process are decoded based on the offset of the reference block by the reference cyclic padding width value, which is determined based on the locations of the luminance samples in the reference block. At least one coded current picture is restored using a cyclic padding process. The syntax elements indicate the offset value for loop processing; or information about the width of the left and right fill. The flag indicates whether syntax elements corresponding to the cyclic padding process are present in the parameter set. The position of a pixel for motion-compensated prediction in a reference image is determined by interpreting the syntax elements corresponding to the cyclic padding process with a clipping process.

EFFECT: increase in the efficiency of coding.

21 cl, 11 dwg

Description

CROSS-REFERENCE TO RELATED APPLICATION

[1] This application claims priority of U.S. Provisional Application No. 62/903,635, filed September 20, 2019, and U.S. Patent Application No. 17/019,692, filed September 14, 2020, which are incorporated herein in their entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[2] This invention generally relates to the field of data processing and, in particular, to encoding and decoding video.

BACKGROUND OF THE INVENTION

[3] Video encoding and decoding using motion-compensated inter-frame prediction has been known for decades. Uncompressed digital video may consist of a series of images, each image having a spatial size of, for example, 1920 x 1080 luma samples and associated chrominance samples. A series of images can have a fixed or variable image rate (informally also known as frame rate), such as 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (resolution 1920x1080 luma samples at 60 Hz) requires bandwidth close to 1.5 Gbps. An hour of such video requires more than 600 GB of disk space.

[4] One of the goals of video encoding and decoding may be to reduce the redundancy of the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, as well as a combination of both, can be used. Lossless compression refers to methods in which an exact copy of the original signal can be restored from a compressed source signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal suitable for the intended application. In the case of video, lossy compression is widely used. The amount of distortion allowed depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of broadcast television applications. The achievable compression ratio may reflect the following: higher distortion resolution/tolerance may result in a higher compression ratio.

[5] The video encoder and video decoder may use techniques from several broad categories including, for example, motion compensation, transformation, quantization, and entropy coding, some of which will be presented below.

[6] The concept of splitting an encoded video bitstream into packets for transport over packet networks has been around for decades. In the beginning, video coding standards and technologies were mostly optimized for bot-centric transport and defined bitstreams. The packetization took place at system level interfaces, as specified, for example, in Real Time Transport Protocol (RTP) payload formats. With the advent of Internet connectivity suitable for mass consumption of video over the Internet, video coding standards reflected this prominent use case through the conceptual differentiation of the video coding layer (VCL) and the network abstraction layer (NAL). NAL units were introduced in H.264 in 2003 and have since been retained in some video coding standards and technologies with minor changes.

[7] A NAL unit can in many cases be considered the smallest entity that a decoder can act upon without necessarily decoding all previous NAL units of a coded video sequence. Thus, NAL units allow the use of certain error-tolerant technologies as well as certain bitstream manipulation techniques, including bitstream reduction using multimedia-enabled network elements (MANEs) such as Selective Forwarding Units (SFUs) or Multipoint Control Units (MCUs). ).

[8] FIG. 1 shows the respective portions of the NAL unit header syntax diagram according to H.264 (101) and H.265 (102) in both cases without any of their respective extensions. In both cases, disabled_zero_bit is a zero bit used to prevent start code emulation in certain system-level environments. The nal_unit_type syntax element refers to the type of data carried by a NAL unit, which may be, for example, one of certain slice types, parameter set types, an extended supplementary information (SEI) message, and so on. The H.265 NAL unit header further contains nuh_layer_id and nuh_temporal_id_plus1, which indicate the spatial/SNR and temporal layer of the encoded picture to which the NAL unit belongs.

[9] It can be seen that the NAL unit header includes only easily parsable fixed length codewords that have no analysis dependency on other data in the bitstream, such as, for example, other NAL unit headers, parameter sets, and so on. Because NAL unit headers are the first octets in a NAL unit, MANE can easily extract, parse, and act on them. In contrast, other high-level syntax elements, such as slice headers or tile headers, are less accessible to MANE because they may require saving the parameter set context and/or processing variable length code points or arithmetically encoded code points.

[10] In addition, it should be noted that the NAL unit headers as shown in FIG. 1 do not include information that can associate a NAL unit with a coded picture that consists of a plurality of NAL units (such as multiple tiles or slices, at least some of which are packed into individual NAL units).

[11] Certain transport technologies such as RTP (RFC 3550), MPEG system standards, ISO file formats, etc. may include certain information, often in the form of time information such as presentation time (in the case of the MPEG file format and ISO) or capture time (in the case of RTP) which can be easily accessed by MANEs and can help link their respective transport units to the encoded pictures. However, the semantics of this information may differ from one transport/storage technology to another and may not be directly related to the image structure used in video encoding. Accordingly, this information may be heuristic at best, and may also not be particularly well suited for determining whether NAL units in a NAL unit stream belong to the same encoded picture.

DISCLOSURE OF THE INVENTION

[12] Embodiments of the invention relate to a method, system, and computer-readable medium for video encoding. According to one aspect, a video encoding method is provided. The method may include decoding the encoded syntax elements corresponding to the wraparound process. At least one encoded current picture is restored using a round robin process. The syntax elements indicate the offset value for loop processing; or information about the width of the left and right padding.

[13] According to another aspect, a computer system for video encoding is provided. A computer system may include one or more processors, one or more computer-readable storage elements, one or more computer-readable tangible storage devices, and program instructions stored on at least one of one or more storage devices for execution by at least one of one or more more processors through at least one of the one or more memory elements, whereby the computer system is capable of performing the method. The method may include decoding the encoded syntax elements corresponding to the wraparound process. At least one encoded current picture is restored using a round robin process. The syntax elements indicate the offset value for loop processing; or information about the width of the left and right padding.

[14] According to another aspect, a computer-readable medium for video encoding is provided. The computer-readable medium may include one or more computer-readable memories and program instructions stored on at least one of the one or more tangible storage devices, the program instructions being executed by a processor. The software instructions are executed by the processor to execute the method, which may suitably include encoded decoding syntax elements corresponding to the wraparound process. At least one encoded current picture is restored using a round robin process. The syntax elements indicate the offset value for loop processing; or information about the width of the left and right padding.

BRIEF DESCRIPTION OF THE DRAWINGS

[15] These and other objects, features and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in conjunction with the accompanying drawings. Various elements of the drawings are not shown to scale, as the illustrations are provided for clarity and ease of understanding by a person skilled in the art, together with a detailed description. On the drawings:

in fig. 1 is a schematic illustration of NAL unit headers according to H.264 and H.265;

in fig. 2 is a schematic illustration of a simplified block diagram of a communication system according to an embodiment;

in fig. 3 is a schematic illustration of a simplified block diagram of a communication system according to an embodiment;

Fig. 4 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment;

Fig. 5 is a schematic illustration of a simplified block diagram of an encoder according to an embodiment;

in fig. 6 is a schematic illustration of syntax elements for offset signaling according to an embodiment;

in fig. 7 is a schematic illustration of syntax elements for encoder pad width signaling according to an embodiment;

in fig. 8 is a schematic illustration of a simplified flowchart of filling processing for each sub-area (tile or sub-image) according to the embodiment;

in fig. 9 is a schematic illustration of syntax elements for signaling a fill offset of each sub-picture, according to an embodiment;

in fig. 10 is a schematic illustration of syntax elements for unified fill offset signaling of each sub-picture according to an embodiment; And

11 is a schematic illustration of a computer system according to an embodiment.

IMPLEMENTATION OF THE INVENTION

[16] Detailed embodiments of the claimed structures and methods are disclosed in this description; however, it can be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which may be embodied in various forms. However, these structures and methods can be embodied in many different forms and should not be construed as being limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this description is complete and complete and fully conveys the scope of legal protection for those skilled in the art. In the description, the details of well-known functions and methods may be omitted in order to avoid unnecessarily obstructing the understanding of the presented embodiments.

[17] Embodiments generally relate to the field of data processing and, in particular, to encoding and decoding video. The following exemplary embodiments described below provide a system, method, and computer program for, among other things, restoring an encoded current picture using a round robin process based on decoded encoded syntax elements. Therefore, some embodiments may improve the computational domain by allowing syntax elements to be encoded to be used in decompressing a compressed image.

[18] As mentioned above, one of the goals of video encoding and decoding may be to reduce the redundancy of the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression, as well as a combination of both, can be used. Lossless compression refers to methods in which an exact copy of the original signal can be restored from a compressed source signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal suitable for the intended application. In the case of video, lossy compression is widely used. The amount of distortion allowed depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of broadcast television applications. The achievable compression ratio may reflect the following: higher distortion resolution/tolerance may result in a higher compression ratio. However, when an image is divided into one or more sub-regions (tile, slice, or sub-image), the edge processing of each sub-region may affect encoding efficiency and subjective visual quality. Adaptive control of edge processing at each sub-region boundary is a key factor in the processing of multimedia data 360. Therefore, it may be preferable to reconstruct the encoded current picture using a wraparound filling process based on the decoding of the encoded syntax elements.

[19] Aspects are described herein with reference to illustrations of block diagrams and/or diagrams of methods, devices (systems), and computer-readable media according to various embodiments. It should be understood that each block of flowchart and/or diagram illustrations, as well as combinations of blocks in flowchart and/or diagram illustrations, may be implemented using computer readable program instructions.

[20] The following exemplary embodiments described below provide a system, method, and computer program that restores an encoded current picture using a round-robin process based on decoded encoded syntax elements.

[21] Figure 2 illustrates a simplified block diagram of a communication system (200) according to an embodiment of the present invention. System (200) may include at least two terminals (210-220) interconnected via network (250). For unidirectional data transmission, the first terminal (210) may encode video data at the local location for transmission to another terminal (220) via the network (250). The second terminal (220) may receive another terminal's encoded video data from the network (250), decode the encoded data, and display the recovered video data. One-way communication may be common in media serving applications and the like.

[22] FIG. 2 shows a second pair of terminals (230, 240) supporting bi-directional encoded video transmission, such as may occur during a video conference. For bi-directional data transmission, each terminal (230, 240) may encode video data captured at a local location for transmission to another terminal over a network (250). Each terminal (230, 240) may also receive the encoded video data transmitted by the other terminal, may decode the encoded data, and may display the recovered video data on a local display device.

[23] In FIG. 2, the terminals (210-240) may be depicted as servers, personal computers, and smartphones, but the principles of the present invention cannot be limited to this. Embodiments of the present invention find application in laptop computers, tablet computers, media players and/or specialized video conferencing equipment. Network (250) is any number of networks that transmit encoded video data between terminals (210-240), including, for example, wired and/or wireless networks. The communications network (250) may communicate on circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks and/or the Internet. For the purposes of this description, the architecture and topology of the network (250) may not be essential to the operation of the present invention unless explained herein below.

[24] Figure 3 illustrates, as an example of application for the disclosed subject matter, the placement of a video encoder and a video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, videoconferencing, digital television, storage of compressed video on digital media including CD, DVD, memory card, etc., and so on. .

[25] The streaming system may include a capture subsystem (313), which may include a video source (301), such as a digital camera, producing, for example, a stream (302) of uncompressed video samples. This stream (302) of samples, depicted with a thick line to emphasize the large amount of data compared to the encoded video bitstreams, can be processed by an encoder (303) connected to the camera (301). The encoder (303) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (304), drawn with a thin line to emphasize the smaller amount of data compared to the sample stream, may be stored on the streaming server (305) for future use. One or more streaming clients (306, 308) can access the streaming server (305) to extract copies (307, 309) of the encoded video bitstream (304). The client (306) may include a video decoder (310) that decodes an incoming copy of the encoded video bitstream (307) and produces an outgoing stream (311) of video samples that can be displayed on a display (312) or other visualization device (not shown). In some streaming systems, streams (304, 307, 309) of video bits may be encoded according to certain video coding/compression standards. Examples of these standards include ITU-T Recommendation H.265. A video coding standard is under development, informally known as Universal Video Coding or VVC. The expanded object can be used in a VVC context.

[26] FIG. 4 may be a functional block diagram of a video decoder (310) according to an embodiment of the present invention.

[27] The receiver (410) may receive one or more encoded video sequences to be decoded by the decoder (310); in the same or another embodiment, one coded video sequence at a time, where the decoding of each coded video sequence is independent of other coded video sequences. An encoded video sequence may be received from a channel (412), which may be a hardware/software connection to a storage device that stores the encoded video data. The receiver (410) may receive encoded video data with other data, such as encoded audio data and/or ancillary data streams, which may be sent to their respective using entities (not shown). The receiver (410) may separate the encoded video sequence from other data. To combat network jitter, a buffer memory (415) can be connected between the receiver (410) and the entropy decoder/parser (420) (hereinafter "parser"). When the receiver (410) receives data from a storage/transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer (415) may not be needed or may be small. For use in best quality packet networks such as the Internet, a buffer (415) may be required, which may be relatively large and may preferably be of an adaptive size.

[28] The video decoder (310) may include a parser (420) for recovering the symbols (421) from the entropy coded video sequence. The categories of these symbols include information used to control the operation of the decoder (310) and potentially information to control a visualization device such as a display (312) that is not an integral part of the decoder but can be connected to it as shown. in Fig.3. The control information for the rendering device(s) may be in the form of additional enhancement information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser (420) may parse/entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be in accordance with a video coding technology or standard, and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The parser (420) may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), and so on. The entropy decoder/parser may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[29] The parser (420) may perform an entropy decoding/parsing operation of the video sequence received from the buffer (415) to create symbols (421).

[30] Symbol recovery (421) may include many different modules depending on the type of encoded video image or parts thereof (such as: inter and intra image, inter and intra block) and other factors. Which modules are involved and how can be controlled by the subgroup control information that has been parsed from the encoded video sequence by the parser (420). The flow of such subgroup control information between the parser (420) and the plurality of modules is not shown below for clarity.

[31] In addition to the functional blocks already mentioned, the decoder 310 can be conceptually subdivided into a number of functional modules, as described below. In a practical implementation operating under commercial constraints, many of these modules interact closely with each other and can be at least partially integrated into each other. However, for purposes of describing the disclosed subject matter, the conceptual division into functional modules below is appropriate.

[32] The first module is the scaling/inverse transform module (451). The scaling/inverse transform module (451) receives the quantized transform coefficient, as well as control information, including which transform to use, block size, quantization factor, quantization scaling matrices, and so on, in the form of symbol(s) (421) from the parser ( 420). It can output blocks containing sample values that can be entered into the aggregator (455).

[33] In some cases, the output samples of the scaling/inverse transform module (451) may refer to an intra-coded block; that is, to a block that does not use prediction information from previously restored images, but can use prediction information from previously restored portions of the current image. Such prediction information may be provided by the intra-frame prediction module (452). In some cases, the intra-frame prediction module (452) generates a block of the same size and shape as the reconstructed block using the already reconstructed surrounding information extracted from the current (partially reconstructed) image (456). The aggregator (455) in some cases adds, for each sample, the prediction information generated by the intra-prediction module (452) to the output sample information provided by the scaling/inverse transform module (451).

[34] In other cases, the output samples of the scaling/inverse transform module (451) may refer to an inter-coded block and potentially a motion-compensated block. In such a case, the motion compensation prediction module (453) may access the reference picture memory (457) to retrieve samples used for prediction. After compensating for the motion of the selected samples according to the symbols (421) associated with the block, these samples can be added by the aggregator (455) to the output of the scaler/demap module (in this case called residual samples or residual signal) to generate output information. readings. Reference picture memory addresses from where the motion compensation prediction module selects prediction samples may be driven by motion vectors available to the motion compensation prediction module in the form of symbols (421), which may have X, Y and reference picture components, for example. Motion compensation may also include interpolation of sample values retrieved from reference picture memory when accurate sub-sample motion vectors are used, motion vector prediction mechanisms, and so on.

[35] The output samples of the aggregator (455) can be subjected to various loop filtering methods in the loop filter module (456). Video compression technologies may include in-loop filtering technologies that are controlled by parameters included in the encoded video bitstream and provided to the loop filtering module (456) as symbols (421) from the parser (420), but may also respond to metainformation received during decoding the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as responding to previously restored and filtered by loop filtering sample values.

[36] The output of the loop filter module (456) may be a stream of samples that can be output to the renderer (312) and also stored in the reference picture memory (456) for use in future inter-frame prediction.

[37] Certain encoded pictures after full recovery can be used as reference pictures for future prediction. Once the encoded picture has been completely reconstructed and the encoded picture has been identified as a reference picture (eg, by parser 420), the current reference picture 456 may become part of the reference picture buffer 457 and fresh current picture memory may be reallocated before the beginning of the reconstruction of the next encoded picture.

[38] Video decoder 420 may perform decoding operations according to a predetermined video compression technology, which may be documented in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax given by the video compression technology or standard used, in the sense that it conforms to the syntax of the video compression technology or standard as specified in the video compression technology document or standard, and in particular their profile document. It may also be necessary for compliance that the complexity of the encoded video sequence be within the boundaries defined by the level of technology or video compression standard. In some cases, levels limit the maximum image size, the maximum frame rate, the maximum recovery sampling rate (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the layers may, in some cases, be further constrained by Hypothetical Reference Decoder (HRD) specifications and metadata for managing the HRD buffer signaled in the encoded video sequence.

[39] In an embodiment, the receiver (410) may receive additional (redundant) data with encoded video. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (420) to correctly decode the data and/or to recover the original video data more accurately. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, and so on.

[40] FIG. 5 may be a functional block diagram of a video encoder (303) according to an embodiment of the present invention.

[41] The encoder (303) may receive video samples from a video source (301) (which is not part of the encoder) that may capture video image(s) for encoding by the encoder (303).

[42] The video source (301) may provide the source video sequence for encoding by the encoder (303) in the form of a digital stream of video samples, which may have any suitable bit depth (eg: 8 bit, 10 bit, 12 bit, ...), any color space (eg BT.601 Y CrCB, RGB, …) and any suitable sample structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (301) may be a storage device storing pre-rendered video. In a videoconferencing system, the video source (303) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual images that, when viewed sequentially, convey motion. The images themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sample structure used, color space, etc. One skilled in the art can easily understand the relationship between pixels and samples. The description below is based on readings.

[43] According to an embodiment, the encoder (303) may encode and compress images of the source video sequence into the encoded video sequence (543) in real time or with any other time constraints as required by the application. Providing an appropriate coding rate is one of the functions of the controller (550). The controller controls other functional modules, as described below, and is functionally associated with these modules. The connection is not shown for clarity. The parameters set by the controller may include parameters related to speed control (picture skip, quantizer, speed-distortion optimization methods lambda value, ...), image size, group of pictures (GOP) layout, maximum motion vector search range, and so on. . A person skilled in the art can easily determine other functions of the controller (550) as they may relate to a video encoder (303) optimized for a particular system design.

[44] Some video encoders operate in what one of ordinary skill in the art would easily recognize as a "coding loop". As a simplified description, an encoding loop may consist of an encoding part of an encoder (530) (hereinafter "source encoder") (responsible for creating symbols based on the input image to be encoded and the reference image(s)), and a (local) decoder (533) built into the encoder (303) that decompresses the symbols to create sample data that the (remote) decoder can also create (because any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies considered in the disclosed subject matter) . This reconstructed sample stream is entered into the reference picture memory 534 . Because decoding a stream of symbols results in bit-accurate results, regardless of the location of the decoder (local or remote), the contents of the reference picture buffer are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction portion of the encoder "sees" as reference picture samples exactly the same sample values that the decoder would "see" using prediction during decoding. This fundamental principle of reference picture synchronism (and the resulting drift if synchronism cannot be maintained due to, for example, channel errors) is well known to those skilled in the art.

[45] The operation of the "local" decoder (533) may be the same as that of the "remote" decoder (310), which has already been described in detail above with reference to Fig.4. Briefly referring also to FIG. 4, however, since the symbols are available and encoding/decoding of the symbols into the encoded video sequence by the entropy encoder (545) and parser (420) can be lossless, the entropy decoding parts of the decoder (310) including the channel (412) , receiver (410), buffer (415), and parser (420) cannot be fully implemented in the local decoder (533).

[46] At this point, it can be observed that any decoding technology other than parsing/entropy decoding that is present in a decoder must also necessarily be present in a substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on the operation of the decoder. The description of encoding technologies can be abbreviated as they are the inverse of fully described decoding technologies. Only in some areas a more detailed description is required, which is given below.

[47] As part of its operation, the source encoder (530) may perform motion-compensated predictive encoding that predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence, which have been designated "reference frames". Thus, the coding engine (532) encodes the differences between the pixel blocks of the input frame and the pixel blocks of the reference frame(s) that can be selected as the prediction reference(s) for the input frame.

[48] The local video decoder (533) may decode the encoded video data of the frames, which may be designated as reference frames, based on the symbols generated by the source encoder (530). The operations of the encoding engine (532) may preferably be lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 5), the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (533) replicates the decoding processes that may be performed by the video decoder on the reference frames and may cause the recovered reference frames to be stored in the reference picture cache (534). Thus, the encoder (303) can locally store copies of the reconstructed reference frames that have common content as reconstructed reference frames to be received by the far-end video decoder (in the absence of transmission errors).

[49] The predictor (535) may perform predictive searches for the encoding engine (532). That is, for a new picture to be encoded, the predictor (535) may search the reference picture memory (534) for sample data (as reference pixel block candidates) or certain metadata such as reference picture motion vectors, block shapes, and so on. further, which can serve as a suitable prediction reference for new images. The predictor (535) may operate on a block-by-pixel basis to find appropriate references for prediction. In some cases, as determined by the search results obtained by the predictor (535), the input image may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (534).

[50] The controller (550) may control the encoding operations of the video encoder (530), including, for example, setting parameters and subgroup parameters used to encode video data.

[51] The output signals of all the above functional modules can be entropy encoded in an entropy encoder (545). An entropy encoder translates symbols generated by various functional modules into an encoded video sequence by lossless symbol compression according to techniques known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, and so on.

[52] The transmitter (540) may buffer the encoded video sequences generated by the entropy encoder (545) to prepare for transmission over a communication channel (560), which may be a hardware/software connection to a storage device that will store the encoded video data. The transmitter (540) may combine the encoded video data from the video encoder (530) with other data to be transmitted, such as encoded audio data and/or ancillary data streams (sources not shown).

[53] The controller (550) may control the operation of the video encoder (303). During encoding, the controller (550) may assign to each encoded picture a certain type of encoded picture, which may affect the encoding methods that can be applied to the corresponding picture. For example, images can often be assigned to one of the following frame types:

[54] An intra-picture (I-picture) may be one that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow various types of intra-images, including, for example, images with independent decoder update. The person skilled in the art will be aware of these variants of I-images and their respective uses and features.

[55] A predictive picture (P-picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most one motion vector and a reference index to predict sample values of each block.

[56] A bidirectional predictive picture (B-picture) may be one that can be encoded and decoded using intra-prediction or inter-prediction using at most two motion vectors and reference indices to predict sample values of each block. Similarly, multi-prediction pictures may use more than two reference pictures and associated metadata to reconstruct a single block.

[57] Source images can typically be spatially divided into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. The blocks may be predictively encoded with reference to other (already encoded) blocks, as determined by the coding assignment applied to the corresponding block images. For example, I-picture blocks may be predictively encoded or they may be predictively encoded with reference to already encoded blocks of the same picture (spatial prediction or intra prediction). P-picture pixel blocks may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one previously encoded reference picture. Blocks of B-pictures may be encoded without prediction, by spatial prediction, or by temporal prediction with reference to one or two previously encoded reference pictures.

[58] The video encoder (303) may perform encoding operations in accordance with a predetermined video coding technology or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore follow the syntax given by the video coding technology or standard being used.

[59] In an embodiment, transmitter (540) may transmit additional data with encoded video. Video encoder (530) may include such data as part of an encoded video sequence. The additional data may comprise temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant images and slices, enhancement additional information (SEI) messages, video usability information (VUI) parameter set fragments, and so on.

[60] In an embodiment, 360-degree video is captured by a set of cameras or a camera device with multiple lenses. Cameras usually cover all directions around the center point of the camera. Images of the same point in time are stitched together, possibly rotated, projected and displayed on the image. Packed images are encoded as encoded into an encoded video bitstream and streamed according to the specific media container file format. The file includes metadata such as projection and packaging information.

[61] In an embodiment, 360-degree video may be projected into 2D video using Equal Rectangular Projection (ERP). The ERP projection can lead to artifacts at the junctions. Advanced ERP Format (PERP) can effectively reduce seam artifacts in rendered viewports that span the left and right boundaries of an ERP image. However, filling and mixing may not be enough to completely solve a seam problem.

[62] In one embodiment, horizontal geometry packing may be used for ERP or PERP to reduce seam artifacts. The padding process for PERP can be the same as for ERP, except that the offset can be based on the width of the unfilled ERP instead of the width of the image to account for the size of the filled areas. If the reference block is outside the left (right) boundary of the reference image, it may be replaced by a "loop" reference block shifted to the right (left) by the width of the ERP. Traditional repeating infill can be used in the vertical direction. Blending the left and right areas with fill is not included in the loop as a post-processing operation.

[63] In an embodiment, the syntax for enabling horizontal reference picture geometry fill for ERP and PERP formats is shown in FIG.

[64] sps_ref_wraparound_enabled_flag (602) equal to 1 indicates that horizontal cyclic motion compensation is used for inter-frame prediction. sps_ref_wraparound_enabled_flag set to 0 indicates that this motion compensation method is not applied.

[65] ref_wraparound_offset(603) specifies an offset in luma samples used to calculate the horizontal position of the transition. ref_wraparound_offset must be greater than pic_width_in_luma_samples - 1, must not be greater than pic_width_in_luma_samples, and must be an integer multiple of MinCbSizeY.

[66] In an embodiment, the syntax for enabling horizontal reference picture geometry fill for ERP and PERP formats is shown in FIG.

[67] sps_ref_wraparound_enabled_flag (702) equal to 1 indicates that horizontal cyclic motion compensation is used for inter-frame prediction. sps_ref_wraparound_enabled_flag set to 0 indicates that this motion compensation method is not applied.

[68] left_wraparound_padding_width (703) Specifies the width of the left padding area in lightness samples. ref_wraparound_offset must be greater than or equal to 0, must not be greater than pic_width_in_luma_samples / 2, and must be an integer multiple of MinCbSizeY.

[69] right_wraparound_padding_width (704) specifies the width of the right padding area in luma samples. ref_wraparound_offset must be greater than or equal to 0, must not be greater than pic_width_in_luma_samples / 2, and must be an integer multiple of MinCbSizeY.

[70] FIG. 8 is a functional block diagram illustrating the steps of a method 900 for encoding video data.

[71] At 901, method 900 includes parsing a set of parameters.

[72] At 902, method 900 includes decoding image division information.

[73] At 903, method 900 includes determining if padding is applied to sub-region boundaries.

[74] At 904, method 900 includes parsing the type of padding if padding is applied to subregion boundaries.

[75] At 905, method 900 includes decoding each sub-region without padding if padding is not applied to sub-region boundaries.

[76] At 906, method 900 includes determining whether a round robin has been applied.

[77] At 907, method 900 includes padding the boundaries of the subregions with loop padding if loop padding is applied.

[78] At 908, method 900 includes padding the boundaries of the subregions with non-cyclic padding if no loop padding is applied.

[79] In an embodiment, the cyclic offset value can be obtained using the following output process:

if ref_wraparound_offset is present

wrapAroundOffset = ref_wraparound_offset

else if left_wraparound_padding_width and right_wraparound_padding_width are present

wrapAroundOffset = pic_width_in_luma_samples - ( left_wraparound_padding_width + right_wraparound_padding_width )

else

wrapAroundOffset = pic_width_in_luma_samples

[80] In an embodiment, to provide horizontal geometric filling of the reference pictures for the ERP and PERP formats, the interpolation processes for the luminance and chrominance samples may be changed.

Clip3(x, y, z) =

ClipH(o, W, x) =

[81] Luminance sample interpolation process. Inputs to this process may include the luma location in whole-sample units (xIntL , yIntL), the luma location in fractional units (xFracL , yFracL), and an array of reference luminance samples refPicLXL. The outputs of this process may include a predicted luminance sample value preSampleLXL.

[82] Variables shift1, shift2 and shift3 are output as follows. The shift1 variable is set to Min (4, BitDepthY - 8), the shift2 variable is set to 6, and the shift3 variable is set to Max (2, 14 - BitDepthY ). The picW variable is set to pic_width_in_luma_samples and the picH variable is set to pic_height_in_luma_samples. The variable xOffset is set to wrapAroundOffset . Luminance interpolation filter coefficients fL[p] can be specified for each 1/16 fractional position p of the sample equal to xFracL or yFracL.

[83] The predicted luminance sample value predSampleLXL is output as follows. If both xFracLand and yFracL are 0, the following applies. If sps_ref_wraparound_enabled_flag is 0, the value of predSampleLXL is output as predSampleLXL = refPicLXL[ Clip3( 0, picW - 1, xIntL ) ][ Clip3( 0, picH - 1, yIntL ) ] << shift3. Otherwise, the value of predSampleLXL is obtained as: predSampleLXL = refPicLXL[ ClipH( xOffset, picW, xIntL ) ][ Clip3( 0, picH - 1, yIntL ) ] << shift3. Otherwise, if xFracL is not equal to 0 and yFracL is equal to 0, the following applies. The value of yPosL is obtained as yPosL = Clip3( 0, picH - 1, yIntL

[84] If sps_ref_wraparound_enabled_flag is 0, the value of predSampleLXL is output as follows:

predSampleLXL = ( fL[ xFracL ][ 0 ] * refPicLXL[ Clip3( 0, picW - 1, xIntL - 3 ) ][ yPosL ] +

fL[ xFracL ][ 1 ] * refPicLXL[ Clip3( 0, picW - 1, xIntL - 2 ) ][ yPosL ] +

fL[ xFracL ][ 2 ] * refPicLXL[ Clip3( 0, picW - 1, xIntL - 1 ) ][ yPosL ] +

fL[ xFracL ][ 3 ] * refPicLXL[ Clip3( 0, picW - 1, xIntL ) ][ yPosL ] +

fL[ xFracL ][ 4 ] * refPicLXL[ Clip3( 0, picW - 1, xIntL + 1 ) ][ yPosL ] +

fL[ xFracL ][ 5 ] * refPicLXL[ Clip3( 0, picW - 1, xIntL + 2 ) ][ yPosL ] +

fL[ xFracL ][ 6 ] * refPicLXL[ Clip3( 0, picW - 1, xIntL + 3 ) ][ yPosL ] +

fL[ xFracL ][ 7 ] * refPicLXL[ Clip3( 0, picW - 1, xIntL + 4 ) ][ yPosL ] ) >> shift1

[85] Otherwise, the value of predSampleLXL is output as follows:

predSampleLXL = ( fL[ xFracL ][ 0 ] * refPicLXL[ ClipH( xOffset, picW, xIntL - 3 ) ][ yPosL ] +

fL[ xFracL ][ 1 ] * refPicLXL[ ClipH( xOffset, picW, xIntL - 2 ) ][ yPosL ] +

fL[ xFracL ][ 2 ] * refPicLXL[ ClipH( xOffset, picW, xIntL - 1 ) ][ yPosL ] +

fL[ xFracL ] [ 3 ] * refPicLXL[ ClipH( xOffset, picW, xIntL ) ][ yPosL ] +

fL[ xFracL ][ 4 ] * refPicLXL[ ClipH( xOffset, picW, xIntL + 1 ) ][ yPosL ] +

fL[ xFracL ][ 5 ] * refPicLXL[ ClipH( xOffset, picW, xIntL + 2 ) ][ yPosL ] +

fL[ xFracL ] [ 6 ] * refPicLXL[ ClipH( xOffset, picW, xIntL + 3 ) ][ yPosL ] +

fL[ xFracL ][ 7 ] * refPicLXL[ ClipH( xOffset, picW, xIntL + 4 ) ][ yPosL ] ) >> shift.

[86] Otherwise, if xFracL is equal to 0 and yFracL is not equal to 0, the value of predSampleLXL is output as follows. If sps_ref_wraparound_enabled_flag is 0, the x PosL value is output as xPosL = Clip3 (0,picW - 1, xIntL ). Otherwise, the xPosL value is output as: xPosL = ClipH (xOffset, picW, xIntL ).

[87] The predicted luminance sample value predSampleLXL is output as follows:

predSampleLXL = ( fL[ yFracL ][ 0 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL - 3 ) ] +

fL[ yFracL ][ 1 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL - 2 ) ] +

fL[ yFracL ][ 2 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL - 1 ) ] +

fL[ yFracL ][ 3 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL ) ] +

fL[ yFracL ][ 4 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL + 1 ) ] +

fL[ yFracL ][ 5 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL + 2 ) ] +

fL[ yFracL ] [ 6 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL + 3 ) ] +

fL[ yFracL ][ 7 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL + 4 ) ] ) >> shift1.

[88] Otherwise, if xFracL is not equal to 0 and yFracL is not equal to 0, the value of predSampleLXL is output as follows. If sps_ref_wraparound_enabled_flag is 0, a sample array temp[n] with n = 0 ..7 is output as follows:

predSampleLXL = (

fL[ yFracL ][ 0 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL - 3 ) ] + fL[ yFracL ][ 1 ] * refPicLXL[ xPosL ]

[ Clip3 ( 0, picH - 1, yIntL - 2 ) ] + fL[ yFracL ][ 2 ] * refPicLXL[ xPosL ]

[ Clip3( 0, picH - 1, yIntL - 1 ) ] + fL[ yFracL ][ 3 ]

* refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL ) ] + fL[ yFracL ][ 4 ]

* refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL + 1 ) ] + fL[ yFracL ][ 5 ]

* refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL + 2 ) ] + fL[ yFracL ]

[ 6 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL + 3 ) ] + fL[ yFracL ]

[ 7 ] * refPicLXL[ xPosL ][ Clip3( 0, picH - 1, yIntL + 4 ) ] ) >> shift1.

[89] Otherwise, the sample array temp[n] with n = 0 ..7 is obtained as follows:

yPosL = Clip3( 0, picH - 1, yIntL + n - 3 )

temp[ n ] = ( fL[ xFracL ][ 0 ] * refPicLXL[ ClipH( xOffset, picW, xIntL - 3 ) ][ yPosL ] +

fL[ xFracL ][ 1 ] * refPicLXL[ ClipH( xOffset, picW, xIntL - 2 ) ][ yPosL ] +

fL[ xFracL ][ 2 ] * refPicLXL[ ClipH( xOffset, picW, xIntL - 1 ) ][ yPosL ] +

fL[ xFracL ][ 3 ] * refPicLXL[ ClipH( xOffset, picW, xIntL ) ][ yPosL ] +

fL[ xFracL ][ 4 ] * refPicLXL[ ClipH( xOffset, picW, xIntL + 1 ) ][ yPosL ] +

fL[ xFracL ][ 5 ] * refPicLXL[ ClipH( xOffset, picW, xIntL + 2 ) ][ yPosL ] +

fL[ xFracL ][ 6 ] * refPicLXL[ ClipH( xOffset, picW, xIntL + 3 ) ][ yPosL ] +

fL[ xFracL ][ 7 ] * refPicLXL[ ClipH( xOffset, picW, xIntL + 4 ) ][ yPosL ] ) >> shift1.

[90] The predicted luminance sample value predSampleLXL is output as follows:

predSampleLXL = ( fL[ yFracL ][ 0 ] * temp[ 0 ] +

fL[ yFracL ][ 1 ] * temp[ 1 ] +

fL[ yFracL ][ 2 ] * temp[ 2 ] +

fL[ yFracL ][ 3 ] * temp[ 3 ] +

fL[ yFracL ][ 4 ] * temp[ 4 ] +

fL[ yFracL ][ 5 ] * temp[ 5 ] +

fL[ yFracL ] [ 6 ] * temp[ 6 ] +

fL[ yFracL ] [ 7 ] * temp[ 7 ] ) >> shift2.

[91] The process of interpolating chrominance samples. The input to this process may include a chromaticity location in full sample units (xIntC, yIntC), a chromaticity location in 1/32 fractional units (xFracC, yFracC), and an array of chrominance reference samples refPicLXC. The outputs of this process may include a predicted luminance sample value predSampleLXL.

[92] Variables shift1, shift2 and shift3 are output as follows. The shift1 variable is set to Min (4, BitDepthY - 8), the shift2 variable is set to 6, and the shift3 variable is set to Max (2, 14 - BitDepthY). The picWC variable is set to pic_width_in_luma_samples / SubWidthC and the picHC variable is set to pic_height_in_luma_samples / SubHeightC. The variable xOffsetC is set to wrapAroundOffset / SubWidthC. Luminance interpolation filter coefficients fC [p] can be specified for each 1/32 fractional position p of a sample equal to xFracC or yFracC.

[93] The predicted chrominance sample value predSampleLXL is output as follows. If both xFracC and yFracC are 0, the following applies. If sps_ref_wraparound_enabled_flag is 0, the value of predSampleLXC is output as:

predSampleLXC =refPicLXC[Clip3 (0,picWC - 1,xIntC )] [Clip3 (0,picHC - 1,yIntC )] << shift3.

[94] Otherwise, the value of predSampleLXC is obtained as predSampleLXC =refPicLXC[ClipH (xOffsetC, picWC,xIntC )] [Clip3 (0,picHC - 1, yIntC) ] << shift3.-

[95] Otherwise, if xFracC is not equal to 0 and yFracC is equal to 0, the following applies. The value of yPosC is derived as yPosC = Clip3( 0, picHC - 1, yIntC ). If sps_ref_wraparound_enabled_flag is 0, the value of predSampleLXC is output as:

predSampleLXC = ( fC[ xFracC ][ 0 ] * refPicLXC[ Clip3( 0, picWC - 1, xIntC - 1 ) ][ yIntC ] +

fC[ xFracC ][ 1 ] * refPicLXC[ Clip3( 0, picWC - 1, xIntC ) ][ yIntC ] +

fC[ xFracC ][ 2 ] * refPicLXC[ Clip3( 0, picWC - 1, xIntC + 1 ) ][ yIntC ] +

fC[ xFracC ][ 3 ] * refPicLXC[ Clip3( 0, picWC - 1, xIntC + 2 ) ][ yIntC ] ) >> shift1.

[96] Otherwise, the value of predSampleLXC is output as:

predSampleLXC = ( fC[ xFracC ][ 0 ] * refPicLXC[ ClipH( xOffsetC, picWC, xIntC - 1 ) ][ yPosC ] +

fC[ xFracC ][ 1 ] * refPicLXC[ ClipH( xOffsetC, picWC, xIntC ) ][ yPosC ] +

fC[ xFracC ][ 2 ] * refPicLXC[ ClipH( xOffsetC, picWC, xIntC + 1 ) ][ yPosC ] +

fC[ xFracC ][ 3 ] * refPicLXC[ ClipH( xOffsetC, picWC, xIntC + 2 ) ][ yPosC ] ) >> shift1.

[97] Otherwise, if xFracL is equal to 0 and yFracL is not equal to 0, the value of predSampleLXL is output as follows. If sps_ref_wraparound_enabled_flag is 0, the xPosC value is output as: xPosC = Clip3 (0,picWC - 1, xIntC) . Otherwise, the xPosC value is obtained as: xPosC = ClipH (xOffsetC,picWC,xIntC )

[98] The predicted chrominance sample value predSampleLXC is obtained as follows:

predSampleLXC = ( fC[ yFracC ][ 0 ] * refPicLXC[ xPosC ][ Clip3( 0, picHC - 1, yIntC - 1 ) ] + fC[ yFracC ][ 1 ] * refPicLXC[ xPosC ][ Clip3( 0, picHC - 1 , yIntC ) ] + fC[ yFracC ][ 2 ] * refPicLXC[ xPosC ][ Clip3( 0, picHC - 1, yIntC +

1 ) ] + fC[ yFracC ][ 3 ] * refPicLXC[ xPosC ][ Clip3( 0, picHC - 1, yIntC + 2 ) ] ) >> shift1.predSampleLXC = (fC[yFracC ]

[0] * refPicLXC[xPosC] [ Clip3 (0,picHC - 1,yIntC - 1)] + fC[yFracC ] [1] *refPicLXC[xPosC ] [Clip3 ( 0,picHC - 1,yIntC )] +

fC [yFracC][2 ] * refPicLXC [xPosC] [ Clip3 (0,picHC - 1,yIntC + 1)] + fC[yFracC ] [3] *refPicLXC[xPosC ] [Clip3 (0 ,picHC - 1,yIntC + 2)]) >> shift1.

[99] Otherwise, if xFracL is equal to 0 and yFracL is not equal to 0, the value of predSampleLXL is output as follows. If sps_ref_wraparound_enabled_flag is 0, the temp[n] countdown array with n = 0 ..3 is output as follows:

yPosC = Clip3( 0, picHC - 1, yIntC + n - 1 )

temp[ n ] = ( fC[ xFracC ][ 0 ] * refPicLXC[ Clip3( 0, picWC - 1, xIntC - 1 ) ][ yPosC ] +

fC[ xFracC ][ 1 ] * refPicLXC[ Clip3( 0, picWC - 1, xIntC ) ][ yPosC ] +

fC[ xFracC ][ 2 ] * refPicLXC[ Clip3( 0, picWC - 1, xIntC + 1 ) ][ yPosC ] +

fC[ xFracC ][ 3 ] * refPicLXC[ Clip3( 0, picWC - 1, xIntC + 2 ) ][ yPosC ] ) >> shift1.

[100] Otherwise, the sample array temp[n] with n = 0 ..3 is output as follows:

yPosC = Clip3( 0, picHC - 1, yIntC + n - 1 )

temp[ n ] = ( fC [ xFracC ][ 0 ] * refPicLXC[ ClipH( xOffsetC, picWC, xIntC - 1 ) ][ yPosC ] +

fC[ xFracC ][ 1 ] * refPicLXC[ ClipH( xOffsetC, picWC, xIntC ) ][ yPosC ] +

fC[ xFracC ][ 2 ] * refPicLXC[ ClipH( xOffsetC, picWC, xIntC + 1 ) ][ yPosC ] +

fC[ xFracC ][ 3 ] * refPicLXC[ ClipH( xOffsetC, picWC, xIntC + 2 ) ][ yPosC ] ) >> shift1.

[101] The predicted chrominance sample value predSampleLXC is output as follows:

predSampleLXC = ( fC[ yFracC ][ 0 ] * temp[ 0 ] +

fC[ yFracC ][ 1 ] * temp[ 1 ] +

fC[ yFracC ][ 2 ] * temp[ 2 ] +

fC[ yFracC ] [ 3 ] * temp[ 3 ] ) >> shift2.

[102] In an embodiment, if sps_ref_wraparound_enabled_flag is equal to 0 or absent, traditional repeated padding may be applied. Otherwise, you can apply circular filling.

[103] In an embodiment, cyclic filling can be applied to both horizontal and vertical boundaries. A flag in a high-level syntax structure can indicate that circular padding is applied both horizontally and vertically.

[104] In an embodiment, circular padding may be applied at the boundaries of tiles or groups of tiles. A flag in a high-level syntax structure can indicate that circular padding is applied both horizontally and vertically.

[105] In an embodiment, the reference picture may be identical to the current picture for motion compensation prediction. Cyclic filling can be applied on the border of the current image when the current image is the reference image.

[106] In an embodiment, when an image is divided into one or more sub-images, which are a rectangular area of one or more slices, the boundary of each sub-image may or may not be treated as an image boundary. Processing a sub-image as an image indicates that the boundary of each sub-image can be filled for motion compensation prediction.

[107] As shown in FIG. 9, in SPS (or any other parameter set), subpic_treated_as_pic_flag [i] equal to 1 indicates that the i-th sub-picture of each encoded picture in CVS is treated as a picture in the decoding process. subpic_treated_as_pic_flag [i] equal to 0 indicates that the i-th sub-picture of each encoded picture in CVS is not treated as a picture in the decoding process. If absent, subpic_treated_as_pic_flag[i] is assumed to be 0.

[108] In the same embodiment, when subpic_treated_as_pic_flag [i] is equal to 1, which indicates that the i-th sub-image is processed as an image, the flag subpic_ref_wraparound_enabled_flag [i] is signaled. subpic_ref_wraparound_enabled_flag [i], equal to 1. subpic_ref_wraparound_enabled_flag [i], equal to 1, indicates that horizontal motion compensation with cyclic filling is applied in inter-frame prediction at the border of the i-th sub-picture of each encoded picture. sps_ref_wraparound_enabled_flag[i] equal to 0 indicates that horizontal wraparound motion compensation is not applied at the boundary of the i-th sub-picture of each encoded picture.

[109] In the same embodiment, subpic_ref_wraparound_offset_minus1[i] plus 1 specifies the offset of the i-th sub-picture of each encoded picture used to calculate the horizontal transition position in units of luminance samples MinCbSizeY.

[110] In the same embodiment, the interpolation filtering process is as follows. Luminance reading interpolation process. Inputs to this process may include a brightness location in full-scale units (xIntL , yIntL ), a brightness location in fractional units (xFracL ,yFracL ), a brightness location in full-scale units (xSbIntL,ySbIntL ), specifying the upper left count of the boundary block to fill in the reference sample relative to the top left brightness sample of the reference image, array of luminance reference samples refPicLXL, half sample interpolation filter index hpelIfIdx, sbWidth variable defining the current subblock width, sbHeight variable defining the height of the current subblock, and brightness location (xSb, ySb) A that defines the top left of the current subblock relative to the top left of the brightness of the current image. The outputs of this process may include a predicted luminance sample value predSampleLXL.

[111] Variables shift1, shift2 and shift3 are output as follows. The shift1 variable is set to Min (4, BitDepthY - 8), the shift2 variable is set to 6, and the shift3 variable is set to Max (2, 14 - BitDepthY ). The picW variable is set to pic_width_in_luma_samples and the picH variable is set to pic_height_in_luma_samples.

[112] The coefficients of the luminance interpolation filter fL[p] for each 1/16 fractional position p of the sample equal to xFracL or yFracL are derived as follows. If MotionModelIdc[xSb][ySb] is greater than 0, and sbWidth and sbHeight are both 4, luma interpolation filter coefficients fL[p] may be specified. Otherwise, the luminance interpolation filter coefficients fL[p] are specified in Table 8-11 depending on hpelIfIdx.

[113] Luminance positions in GS units (xInti, yInti ) are derived as follows for i = 0… 7. If subpic_treated_as_pic_flag [SubPicIdx] is 1, the following applies:

xInti = Clip3( SubPicLeftBoundaryPos, SubPicRightBoundaryPos, subpic_ref_wraparound_enabled_flag[ SubPicIdx ] ClipH( ( subpic_ref_wraparound_offset_minus1[ SubPicIdx ] + 1 ) * MinCbSizeY, SubPicRightBoundaryPos, xIntL + i - 3 ) :

xIntL + i - 3 )

.yInti = Clip3( SubPicTopBoundaryPos, SubPicBotBoundaryPos, yIntL + i - 3 ).

[114] Otherwise (subpic_treated_as_pic_flag [SubPicIdx] is 0), the following applies:

xInti = Clip3( 0, picW - 1, sps_ref_wraparound_enabled_flag

ClipH( ( sps_ref_wraparound_offset_minus1 + 1 ) * MinCbSizeY, picW, xIntL + i - 3 ) :

xIntL + i - 3 )

yInti = Clip3( 0, picH - 1, yIntL + i - 3 )

[115] The location of the brightness in full sample units is further changed as follows for i = 0…7:

xInti = Clip3( xSbIntL - 3, xSbIntL + sbWidth + 4, xInti )

yInti = Clip3( ySbIntL - 3, ySbIntL + sbHeight + 4, yInti )

[116] The predicted luminance sample value predSampleLXL is output as follows. If both xFracLand yFracL are 0 , the value of predSampleLXL is output as: predSampleLXL

= refPicLXL [xInt3][ yInt3]< < shift3.

> > shift1.[117] Otherwise, if xFracL is not equal to 0 and yFracL is equal to 0, the value of predSampleLXL is output as predSampleLXL =

>> shift1.

> > shift1.[118] Otherwise, if xFracL is 0 and yFracL is not 0, the value of predSampleLXL is output as predSampleLXL =

>> shift1.

[119] Otherwise, if xFracL is not equal to 0 and yFracL is not equal to 0, the value of predSampleLXL is output as follows.

>> shift1.[120] Sample array temp[n] with n = 0 ..7, obtained as: temp[n] =

>>shift1.

>> shift2.[121] The predicted luminance sample value predSampleLXL is output as predSampleLXL=

>>shift2.

[122] In the same or different embodiment, as shown in FIG. 10, in SPS (or any other parameter set), all_subpic_treated_as_pic_flag equal to 1 indicates that any subpicture of each encoded picture in CVS is treated as a picture in the decoding process. subpic_treated_as_pic_flag[i] equal to 0 indicates that any subpicture of each encoded picture in CVS is not treated as a picture during the decoding process.

[123] In the same embodiment, when all_subpic_treated_as_pic_flag [i] is equal to 1, the all_subpic_ref_wraparound_enabled_flag flag is signaled. all_subpic_ref_wraparound_enabled_flag equal to 1 indicates that horizontal cyclic motion compensation is applied in inter-picture prediction at the boundary of any sub-picture of each encoded picture. all_sps_ref_wraparound_enabled_flag equal to 0 indicates that horizontal circular motion compensation is not applied on the boundary of any sub-picture of each encoded picture.

[124] In the same embodiment, all_subpic_ref_wraparound_offset_minus1[i] plus 1 specifies the offset of any subimage of each encoded image used to calculate the horizontal transition position in units of MinCbSizeY luma samples. In the same embodiment, when subpic_treated_as_pic_flag[i] is absent, the value of subpic_treated_as_pic_flag[i] is assumed to be all_subpic_treated_as_pic_flag.

[125] In the same embodiment, when subpic_ref_wraparound_enabled_flag[i] is absent, the value of subpic_ref_wraparound_enabled_flag[i] is assumed to be all_subpic_ref_wraparound_enabled_flag.

[126] In the same embodiment, when subpic_ref_wraparound_offset_minus1[i] is present, the value of subpic_ref_wraparound_offset_minus1[i] is assumed to be all_subpic_ref_wraparound_offset_minus1.

[127] The wraparound process methods described above may be implemented in computer software using computer readable instructions and physically stored on one or more computer readable media. For example, FIG. 11 illustrates a computer system 800 suitable for implementing certain embodiments of the disclosed subject matter.

[128] Computer software may be coded using any suitable machine code or computer language, which may be the object of assembly, compilation, linking, or similar mechanisms to produce code containing instructions that can be executed directly or through interpretation, execution of microcode, and and etc., computer central processing units (CPUs), graphics processing units (GPUs), etc.

[129] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[130] The components shown in FIG. 11 for computer system 800 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present invention. The component configuration is also not to be interpreted as having any dependency or requirement relating to any one or combination of the components illustrated in the exemplary embodiment of computer system 800.

[131] Computer system 800 may include certain human-machine interface input devices. Such a human-machine interface input device may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a control glove), audio input (e.g., voice, handclaps), visual input (for example: gestures), olfactory input (not shown). HMI devices can also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, ambient sound), images (e.g. scanned images, photographic images taken from a camera for still images), video (eg 2D video, 3D video including stereoscopic video).

[132] Human interface input devices may include one or more of (only one of each depicted): keyboard 801, mouse 802, trackpad 803, touch screen 810, data glove 804, joystick 805, microphone 806, scanner 807, camera 808.

[133] Computer system 800 may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human interface output devices may include tactile output devices (e.g., haptic feedback from a touch screen (810), control glove 804, or joystick (805), but there may also be tactile feedback devices that do not serve to as input devices), audio output devices (for example: speakers (809), headphones (not shown)), image output devices (for example, screens (810), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without haptic feedback capability - some of which may be capable of 2D visual output or more than 3D output by means such as stereographic output; virtual glasses realities (not shown), holographic displays and smoke tanks (not shown) and printers (not shown).

[134] The computer system (800) may also include human accessible storage devices and associated media such as optical media including CD/DVD ROM/RW (820) with CD/DVD or similar media (821), flash drive (822), removable hard drive or solid state drive (823), legacy magnetic media such as tape and floppy disk (not shown), dedicated ROM/ASIC/PLD based devices such as dongles (not shown), and etc.

[135] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other time signals.

[136] Computer system 800 may also include an interface to one or more communication networks. Networks can be, for example, wireless, wired, optical. In addition, networks can be LAN, WAN, metropolitan, automotive, industrial, real-time, delay tolerant, etc. Examples of networks include LANs such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired TV or wireless wide area digital networks, including cable TV, satellite TV and terrestrial TV, automotive and industrial equipment, including CAN bus, etc. Some networks usually require external network interface adapters that are connected to certain general purpose data ports or peripheral buses (849) (eg, USB ports of the computer system 800); others are typically integrated into the core of computer system 800 by being attached to a system bus as described below (eg, an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 800 can communicate with other entities. Such communication can be unidirectional, receive-only (for example, broadcast television), unidirectional, send-only (for example, CAN bus to certain CAN-bus devices), or bidirectional, for example, to other computer systems using a local or wide area digital network. . Each of these networks and network interfaces may use specific protocols and protocol stacks as described above.

[137] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core 840 of the computer system 800.

[138] The core (840) may include one or more central processing units (CPUs) (841), graphics processing units (GPUs) (842), dedicated field programmable processors in the form of field programmable gate arrays (FPGAs) (843), hardware accelerators (844) for specific tasks, etc. These devices, along with read-only memory (ROM) (845), random access memory (846), internal storage devices such as internal non-user-accessible hard drives, solid state drives, and etc. (847) can be connected via the system bus (848). In some computer systems, the 848 system bus may be available as one or more physical connectors to allow expansion with additional CPUs, GPUs, etc. Peripherals may be connected either directly to the kernel system bus 848 or via the 849 peripheral bus. .Peripheral bus architecture includes PCI, USB, etc.

[139] CPU 841, GPU 842, FGPA 843, and accelerators 844 may execute certain instructions, which in combination may constitute the aforementioned computer code. This computer code may be stored in ROM 845 or RAM 846. Transient data may also be stored in RAM 846, while persistent data may be stored in, for example, internal mass storage 847. Fast storage and retrieval to any of the storage devices may be accessed by the use of cache memory, which may be closely associated with one or more CPU 841, GPU 842, mass storage 847, ROM 845, RAM 846, etc.

[140] The computer-readable medium may contain computer code for performing various computer-implemented operations. The media and computer code may be specially designed and constructed for the purposes of the present invention, or they may be well known and available to those skilled in the art of computer software.

[141] By way of example, and not limitation, a computer system having architecture 800, and in particular core 840, may provide functionality through processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied on one or more tangible machine-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as certain core storage devices 840 that are of a non-transitory nature, such as internal mass storage 847 or ROM 845. implementation of the present invention may be stored in such devices and executed by the core 840. The computer-readable medium may include one or more memory devices or chips in accordance with specific needs. The software may direct the core 840, and in particular the processors therein (including CPUs, GPUs, FPGAs, and the like), to execute certain processes or certain portions of the specific processes described herein, including the definition of data structures stored in RAM 846 , and changing such data structures in accordance with the processes defined by the software. In addition, or alternatively, a computer system may provide functionality as a result of logic embedded in hardware circuitry or otherwise embodied in circuitry (e.g., an 844 accelerator) that can operate in place of or in conjunction with software to perform certain processes or parts of specific processes described in this document. The link to the software may include logic and vice versa where applicable. A reference to a computer-readable medium may include circuitry (eg, an integrated circuit (IC)) storing software to execute, circuitry embodying logic to execute, or both, where applicable. The present invention includes any suitable combination of hardware and software.

[142] Although this description discloses several exemplary embodiments, there are variations, permutations, and various replacement equivalents that fall within the scope of the invention. Thus, it will be appreciated that those skilled in the art will be able to develop numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the invention and thus are within its spirit and scope.

Claims (29)

1. A video decoding method executed by a processor, including the following steps: decoding the encoded syntax elements corresponding to the cyclic fill process based on the offset of the reference block by the reference cyclic fill width value, which is determined based on the locations of the luminance samples in the reference block; And restoring at least one encoded current picture using a round robin process. 2. The method of claim 1, wherein the decoded syntax elements indicate an offset value for loop processing; or information about the width of the left and right padding. 3. The method of claim 2, wherein the flag indicates whether syntax elements corresponding to the wraparound process are present in the parameter set. 4. The method of claim 3, wherein the pixel position for motion-compensated prediction in the reference image is determined by interpreting syntax elements corresponding to the round-robin process with a clipping process. 5. The method of claim 4, wherein the fractional pixel is interpolated for motion compensation prediction based on the determined pixel position. 6. The method of claim 5, wherein the looped region is blended by blending the left and right filled areas of the looped region to remove the seam artifact as a post-processing. 7. The method of claim. 1, in which the process of cyclic filling is applied at the border of a tile or group of tiles. 8. The method of claim 1, wherein at least one encoded current picture is a reference for motion compensation. 9. The method of claim 8, wherein the wrapping process is applied at a boundary of at least one encoded current picture. 10. The method of claim 1, wherein the cycling process is applied to the horizontal boundary, the vertical boundary, or both vertical and horizontal boundaries. 11. The method of claim 10, wherein the flag in the high-level syntax structure indicates whether the wraparound process is applied on a horizontal boundary, a vertical boundary, or both vertical and horizontal boundaries. 12. The method of claim 1, wherein the cycling process is applied at sub-image boundaries. 13. A computer system for video encoding, comprising: one or more computer-readable non-transitory storage media configured to store computer program code; And one or more computer processors configured to access said computer program code and operate in accordance with the instructions of said computer program code, said computer program code including: decoding code configured to cause one or more computer processors to decode encoded syntax elements corresponding to the cyclic fill process based on a reference block offset by a reference cyclic fill width value that is determined based on luminance sample locations within the reference block; And recovery code configured to cause one or more computer processors to recover at least one encoded current picture using a wraparound process. 14. The computer system according to claim 13, wherein the decoded syntax elements indicate an offset value for loop processing; or information about the width of the left and right padding. 15. The computer system of claim 14, wherein the flag indicates whether syntax elements corresponding to the wrapping process are present in the parameter set. 16. The computer system of claim 15, wherein the pixel position for motion compensation prediction in the reference image is determined by interpreting the syntax elements corresponding to the wraparound process with a clipping process. 17. The computer system of claim 16, wherein the fractional pixel is interpolated for motion compensation prediction based on the determined pixel position. 18. The computer system of claim 17, wherein the looped region is blended by blending left and right filled areas of the looped area to remove seam artifact as post-processing. 19. The computer system of claim 13, wherein the cycling process is applied at the boundary of a tile or group of tiles. 20. The computer system of claim 13, wherein at least one encoded current picture is a reference for motion compensation. 21. A non-transitory computer-readable medium storing a video encoding computer program, the computer program configured to cause one or more computer processors to perform: decoding the encoded syntax elements corresponding to the cyclic fill process based on the offset of the reference block by the cyclic fill width reference value, which is determined based on the locations of the luminance samples in the reference block; And restoring at least one encoded current picture using a round robin process.

Images