CN116868571A

CN116868571A - Improved local illumination compensation for inter prediction

Info

Publication number: CN116868571A
Application number: CN202280016220.6A
Authority: CN
Inventors: 朱弘正; 陈漪纹; 修晓宇; 陈伟; 郭哲玮; 王祥林; 于冰
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2021-02-22
Filing date: 2022-02-22
Publication date: 2023-10-10
Also published as: WO2022178433A1

Abstract

An electronic device performs a method of decoding a video signal. The method comprises the following steps: determining two or more reference sample point pairs, each of the reference sample point pairs comprising an adjacent reconstructed luma sample point of the current block and a corresponding adjacent reconstructed luma sample point of the reference block; classifying the two or more reference sample point pairs into one or more groups; deriving one or more linear models based on the classified one or more group sample point pairs; and predicting luma sample values in the current block by applying the one or more linear models to corresponding reconstructed luma samples in the reference block. In some embodiments, the reference block is derived from the current block by a motion vector shift.

Description

Improved local illumination compensation for inter prediction

Cross Reference to Related Applications

The present application is based on and claims priority from U.S. provisional application No. 63/152,273 entitled "Improved local illumination compensation for inter prediction," filed on 22 nd month 2021, the entire contents of which are incorporated herein by reference.

Technical Field

The application relates to video codec and compression. More particularly, the present application relates to methods and apparatus for improving codec efficiency and simplifying the complexity of local illumination compensation (local illumination compensation, LIC).

Background

Various electronic devices (e.g., digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc.) support digital video. The electronic device sends and receives or otherwise communicates digital video data over a communication network and/or stores the digital video data on a storage device. Because of the limited bandwidth capacity of the communication network and the limited storage resources of the storage device, video data may be compressed using video codec according to one or more video codec standards before it is transmitted or stored. For example, video coding standards include general video coding (VVC), joint exploration test model (JEM), high efficiency video coding (HEVC/h.265), advanced video coding (AVC/h.264), moving Picture Experts Group (MPEG) coding, and so forth. Video coding and decoding typically employ prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploit redundancy inherent in video data. Video codec aims at compressing video data into a form using a lower bit rate while avoiding or minimizing degradation of video quality.

Disclosure of Invention

Embodiments are described in connection with video data encoding and decoding, and in particular, embodiments relate to methods and apparatus for improving codec efficiency and simplifying the complexity of local illumination compensation (local illumination compensation, LIC).

According to a first aspect of the present application, a method of decoding a video signal, comprises: determining two or more reference sample point pairs, each of the reference sample point pairs comprising an adjacent reconstructed luma sample point of the current block and a corresponding adjacent reconstructed luma sample point of the reference block; classifying the two or more reference sample point pairs into one or more groups; deriving one or more linear models based on the classified one or more group sample point pairs; and predicting luma sample values in the current block by applying the one or more linear models to corresponding reconstructed luma samples in the reference block. In some embodiments, the reference block is derived from the current block by a motion vector shift.

According to a second aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. The programs, when executed by one or more processing units, cause the electronic device to perform the method of decoding a video signal as described above.

According to a third aspect of the application, a non-transitory computer readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units. The programs, when executed by one or more processing units, cause the electronic device to perform the method of decoding a video signal as described above.

According to a fourth aspect of the present application, a computer readable storage medium has stored therein a bitstream comprising video information generated by a method of decoding a video signal as described above.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.

Fig. 2 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

Fig. 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

Fig. 4A-4E are block diagrams illustrating how frames are recursively partitioned into multiple video blocks of different sizes and shapes according to some embodiments of the present disclosure.

Fig. 5 is a block diagram illustrating an exemplary straight-line derivation of parameters (α and β) using a min-max method according to some embodiments of the present disclosure.

Fig. 6 is a block diagram illustrating exemplary locations of samples for deriving parameters (α and β) according to some embodiments of the present disclosure.

Fig. 7 is a block diagram illustrating exemplary derivation of parameters for a multi-model linear model (MMLM) by classifying neighboring samples into two groups, according to some embodiments of the present disclosure.

Fig. 8 is a block diagram illustrating exemplary derivation of prediction model parameters by classifying neighboring samples into two groups based on a corner point, according to some embodiments of the present disclosure.

Fig. 9A and 9B are block diagrams illustrating exemplary neighboring samples for deriving Illumination Compensation (IC) parameters according to some embodiments of the present disclosure.

Fig. 10 is a block diagram illustrating exemplary locations of reference samples (in gray blocks) in method 1 of table 4 according to some embodiments of the present disclosure.

Fig. 11 is a block diagram illustrating exemplary locations of reference samples (in gray blocks) in method 2 of table 4 according to some embodiments of the present disclosure.

Fig. 12 is a block diagram illustrating exemplary locations of reference samples (in gray blocks) in method 3 of table 4 according to some embodiments of the present disclosure.

Fig. 13 is a block diagram illustrating exemplary locations of reference samples (in gray blocks) in method 4 of table 4 according to some embodiments of the present disclosure.

Fig. 14 is a flowchart illustrating an exemplary process of decoding a video signal according to some embodiments of the present disclosure.

FIG. 15 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure.

Detailed Description

Reference will now be made in detail to the present embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth in order to provide an understanding of the subject matter presented herein. It will be apparent, however, to one of ordinary skill in the art that various alternatives may be used and that the subject matter may be practiced without these specific details without departing from the scope of the claims. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the figures are used for distinguishing between objects and not for describing any particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate, such that embodiments of the disclosure described herein may be implemented in other sequences than those illustrated in the figures or otherwise described in the disclosure.

Fig. 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. As shown in fig. 1, the system 10 includes a source device 12, the source device 12 generating and encoding video data to be later decoded by a target device 14. The source device 12 and the target device 14 may comprise any of a wide variety of electronic devices including desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video gaming machines, video streaming devices, and the like. In some implementations, the source device 12 and the target device 14 are equipped with wireless communication capabilities.

In some implementations, the target device 14 may receive encoded video data to be decoded via the link 16. Link 16 may comprise any type of communication medium or device capable of moving encoded video data from source device 12 to destination device 14. In one example, link 16 may include a communication medium that enables source device 12 to transmit encoded video data directly to destination device 14 in real-time. The encoded video data may be modulated according to a communication standard (e.g., a wireless communication protocol) and transmitted to the target device 14. The communication medium may include any wireless or wired communication medium such as a Radio Frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the internet. The communication medium may include routers, switches, base stations, or any other means that may be advantageous to facilitate communication from source device 12 to destination device 14.

In other embodiments, encoded video data may be sent from output interface 22 to storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28. Storage device 32 may include any of a variety of distributed or locally accessed data storage media such as a hard drive, blu-ray disc, digital Versatile Disc (DVD), compact disc read only memory (CD-ROM), flash memory, volatile or nonvolatile memory, or any other suitable digital storage media for storing encoded video data. In another example, storage device 32 may correspond to a file server or another intermediate storage device that may hold encoded video data generated by source device 12. The target device 14 may access the stored video data from the storage device 32 via streaming or download. The file server may be any type of computer capable of storing and transmitting encoded video data to the target device 14. Exemplary file servers include web servers (e.g., for web sites), file Transfer Protocol (FTP) servers, network Attached Storage (NAS) devices, or local disk drives. The target device 14 may access the encoded video data through any standard data connection suitable for accessing encoded video data stored on a file server, including a wireless channel (e.g., a wireless fidelity (Wi-Fi) connection), a wired connection (e.g., digital Subscriber Line (DSL), cable modem, etc.), or a combination of both a wireless channel and a wired connection. The transmission of encoded video data from storage device 32 may be streaming, download, or a combination of both streaming and download.

As shown in fig. 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as the following or a combination of such sources: a video capture device (e.g., a video camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and/or a computer graphics system for generating computer graphics data as source video. As one example, if video source 18 is a video camera of a security monitoring system, source device 12 and target device 14 may form a camera phone or video phone. However, the embodiments described in this disclosure may be generally applicable to video codecs and may be applied to wireless and/or wired applications.

Captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to the target device 14 via the output interface 22 of the source device 12. The encoded video data may also (or alternatively) be stored on the storage device 32 for later access by the target device 14 or other device for decoding and/or playback. Output interface 22 may further include a modem and/or a transmitter.

The target device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and/or modem and receives encoded video data over link 16. The encoded video data transmitted over link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included in encoded video data transmitted over a communication medium, stored on a storage medium, or stored on a file server.

In some implementations, the target device 14 may include a display device 34, and the display device 34 may be an integrated display device and an external display device configured to communicate with the target device 14. Display device 34 displays decoded video data to a user and may comprise any of a variety of display devices, such as a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or another type of display device.

Video encoder 20 and video decoder 30 may operate in accordance with proprietary standards or industry standards (e.g., section 10 of VVC, HEVC, MPEG-4, AVC) or extensions of such standards. It should be appreciated that the present application is not limited to a particular video encoding/decoding standard and may be applicable to other video encoding/decoding standards. It is generally contemplated that video encoder 20 of source device 12 may be configured to encode video data according to any of these current or future standards. Similarly, it is also generally contemplated that the video decoder 30 of the target device 14 may be configured to decode video data according to any of these current or future standards.

Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder and/or decoder circuits, such as one or more microprocessors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), discrete logic devices, software, hardware, firmware or any combinations thereof. When implemented in part in software, the electronic device can store instructions for the software in a suitable non-transitory computer readable medium and execute the instructions in hardware using one or more processors to perform the video encoding/decoding operations disclosed in the present disclosure. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder/decoder (CODEC) in the respective device.

Fig. 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in this disclosure. Video encoder 20 may perform intra-prediction encoding and inter-prediction encoding on video blocks within video frames. Intra-prediction encoding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-prediction encoding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding, the term "frame" may be used as a synonym for the term "image" or "picture".

As shown in fig. 2, video encoder 20 includes a video data memory 40, a prediction processing unit 41, a Decoded Picture Buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy encoding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra prediction processing unit 46, and an intra Block Copy (BC) unit 48. In some implementations, video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63, such as a deblocking filter, may be located between adder 62 and DPB 64 to filter block boundaries to remove blocking artifacts from the reconstructed video. In addition to the deblocking filter, another loop filter, such as a Sample Adaptive Offset (SAO) filter and/or an Adaptive Loop Filter (ALF), may be used to filter the output of adder 62. In some examples, the loop filter may be omitted and the decoded video block may be provided directly to DPB 64 by adder 62. Video encoder 20 may take the form of fixed or programmable hardware units, or may be dispersed in one or more of the fixed or programmable hardware units described.

Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data store 40 may be obtained, for example, from video source 18 shown in fig. 1. DPB 64 is a buffer that stores reference video data (e.g., reference frames or pictures) for use by video encoder 20 (e.g., in intra-prediction encoding mode or inter-prediction encoding mode) in encoding video data. Video data memory 40 and DPB 64 may be formed from any of a variety of memory devices. In various examples, video data memory 40 may be on-chip with other components of video encoder 20, or off-chip with respect to those components.

As shown in fig. 2, after receiving the video data, a dividing unit 45 within the prediction processing unit 41 divides the video data into video blocks. This partitioning may also include partitioning the video frame into slices, tiles (tiles) (e.g., a set of video blocks), or other larger Coding Units (CUs) according to a predefined split structure (e.g., a Quadtree (QT) structure) associated with the video data. A video frame may be considered a two-dimensional array or matrix of samples having sample values. The samples in the array may also be referred to as pixels or picture elements (pels). The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and/or resolution of the video frame. For example, a video frame may be divided into a plurality of video blocks by using QT segmentation. A video block may also be considered as a two-dimensional array or matrix of samples having sample values, but with dimensions smaller than those of a video frame. The number of samples in the horizontal and vertical directions (or axes) of the video block defines the size of the video block. The video block may be further partitioned into one or more block partitions or sub-blocks (which may again form blocks) by, for example, iteratively using QT partitioning, binary Tree (BT) partitioning, or Trigeminal Tree (TT) partitioning, or any combination thereof. It should be noted that the term "block" or "video block" as used herein may be a portion of a frame or picture, especially a rectangular (square or non-square) portion. Referring to HEVC and VVC, a block or video block may be or correspond to a Coding Tree Unit (CTU), a CU, a Prediction Unit (PU), or a Transform Unit (TU) and/or may be or correspond to a respective block (e.g., a Coding Tree Block (CTB), a Coding Block (CB), a Prediction Block (PB), or a Transform Block (TB)) and/or sub-block.

The prediction processing unit 41 may select one of a plurality of possible prediction coding modes, for example, one of one or more inter prediction coding modes of a plurality of intra prediction coding modes, for the current video block based on the error result (e.g., the coding rate and the distortion level). The prediction processing unit 41 may provide the resulting intra-or inter-prediction encoded block to the adder 50 to generate a residual block and to the adder 62 to reconstruct the encoded block for subsequent use as part of a reference frame. Prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra mode indicators, partition information, and other such syntax information) to entropy encoding unit 56.

To select an appropriate intra-prediction encoding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction encoding of the current video block with respect to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-prediction encoding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, selecting an appropriate encoding mode for each block of video data.

In some embodiments, motion estimation unit 42 determines the inter-prediction mode for the current video frame by generating a motion vector from a predetermined pattern within the sequence of video frames, the motion vector indicating a displacement of a video block within the current video frame relative to a predicted block within a reference video frame. The motion estimation performed by the motion estimation unit 42 is a process of generating a motion vector that estimates motion for a video block. For example, the motion vector may indicate the displacement of a video block within a current video frame or picture relative to a predicted block within a reference frame associated with the current block being encoded within the current frame. The predetermined pattern may designate video frames in the sequence as P-frames or B-frames. The intra BC unit 48 may determine the vector (e.g., block vector) for intra BC encoding in a similar manner as the motion vector used for inter prediction by the motion estimation unit 42, or may determine the block vector using the motion estimation unit 42.

The predicted block of the video block may be or may correspond to a block or reference block of a reference frame that is considered to closely match the video block to be encoded in terms of pixel differences, which may be determined by Sum of Absolute Differences (SAD), sum of Squared Differences (SSD), or other difference metrics. In some implementations, video encoder 20 may calculate values for sub-integer pixel positions of reference frames stored in DPB 64. For example, video encoder 20 may interpolate values for one-quarter pixel positions, one-eighth pixel positions, or other fractional pixel positions of the reference frame. Accordingly, the motion estimation unit 42 can perform motion search with respect to the full pixel position and the fractional pixel position and output a motion vector having fractional pixel accuracy.

The motion estimation unit 42 calculates a motion vector for a video block in an inter prediction encoded frame by: the location of the video block is compared to the location of the predicted block of the reference frame selected from the first reference frame list (list 0) or the second reference frame list (list 1), each of which identifies one or more reference frames stored in DPB 64. The motion estimation unit 42 sends the calculated motion vector to the motion compensation unit 44 and then to the entropy encoding unit 56.

The motion compensation performed by the motion compensation unit 44 may involve acquiring or generating a prediction block based on the motion vector determined by the motion estimation unit 42. Upon receiving the motion vector for the current video block, motion compensation unit 44 may locate the prediction block to which the motion vector points in one of the reference frame lists, obtain the prediction block from DPB 64, and forward the prediction block to adder 50. Adder 50 then forms a residual video block of pixel differences by subtracting the pixel values of the prediction block provided by motion compensation unit 44 from the pixel values of the current video block being encoded. The pixel differences forming the residual video block may include a luma difference component or a chroma difference component or both. Motion compensation unit 44 may also generate syntax elements associated with the video blocks of the video frames for use by video decoder 30 in decoding the video blocks of the video frames. The syntax elements may include, for example, syntax elements defining motion vectors used to identify the prediction block, any flags indicating the prediction mode, or any other syntax information described herein. It should be noted that the motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes.

In some embodiments, the intra BC unit 48 may generate vectors and obtain prediction blocks in a manner similar to that described above in connection with the motion estimation unit 42 and the motion compensation unit 44, but these prediction blocks are in the same frame as the current block being encoded, and these vectors are referred to as block vectors rather than motion vectors. In particular, the intra BC unit 48 may determine an intra prediction mode to be used to encode the current block. In some examples, intra BC unit 48 may encode the current block using various intra prediction modes, e.g., during separate encoding passes, and test their performance through rate-distortion analysis. Next, the intra BC unit 48 may select an appropriate intra prediction mode from among the various tested intra prediction modes to use and generate the intra mode indicator accordingly. For example, the intra BC unit 48 may calculate rate distortion values for various tested intra prediction modes using rate distortion analysis, and select the intra prediction mode having the best rate distortion characteristics among the tested modes to use as the appropriate intra prediction mode. Rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and an original uncoded block that is coded to produce the coded block, as well as the bit rate (i.e., number of bits) used to produce the coded block. The intra BC unit 48 may calculate ratios from the distortion and rate for the various encoded blocks to determine which intra prediction mode exhibits the best rate distortion value for the block.

In other examples, intra BC unit 48 may use motion estimation unit 42 and motion compensation unit 44, in whole or in part, to perform such functions for intra BC prediction in accordance with embodiments described herein. In either case, for intra block copying, the prediction block may be a block that is considered to closely match the block to be encoded in terms of pixel differences, which may be determined by SAD, SSD, or other difference metric, and identifying the prediction block may include calculating a value for a sub-integer pixel location.

Regardless of whether the prediction block is from the same frame according to intra prediction or from a different frame according to inter prediction, video encoder 20 may form the residual video block by subtracting the pixel values of the prediction block from the pixel values of the current video block being encoded. The pixel differences forming the residual video block may include both luma component differences and chroma component differences.

As an alternative to inter prediction performed by the motion estimation unit 42 and the motion compensation unit 44 or intra block copy prediction performed by the intra BC unit 48 as described above, the intra prediction processing unit 46 may intra-predict the current video block. In particular, intra-prediction processing unit 46 may determine an intra-prediction mode for encoding the current block. To this end, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction processing unit 46 (or a mode selection unit in some examples) may select an appropriate intra-prediction mode from the tested intra-prediction modes for use. Intra-prediction processing unit 46 may provide information to entropy encoding unit 56 indicating the intra-prediction mode selected for the block. Entropy encoding unit 56 may encode information into the bitstream that indicates the selected intra-prediction mode.

After the prediction processing unit 41 determines a prediction block for the current video block via inter prediction or intra prediction, the adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform, such as a Discrete Cosine Transform (DCT) or a conceptually similar transform.

The transform processing unit 52 may send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficient to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The quantization level may be modified by adjusting the quantization parameter. In some examples, quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.

After quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context Adaptive Variable Length Coding (CAVLC), context Adaptive Binary Arithmetic Coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability Interval Partition Entropy (PIPE) coding, or another entropy encoding method or technique. The encoded bitstream may then be sent to the video decoder 30 as shown in fig. 1 or archived in the storage device 32 as shown in fig. 1 for later transmission to the video decoder 30 or retrieval by the video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transforms, respectively, to reconstruct the residual video block in the pixel domain for generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values for use in motion estimation.

Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used by the intra BC unit 48, the motion estimation unit 42, and the motion compensation unit 44 as a prediction block to inter-predict another video block in a subsequent video frame.

Fig. 3 is a block diagram illustrating an exemplary video decoder 30 according to some embodiments of the present disclosure. Video decoder 30 includes video data memory 79, entropy decoding unit 80, prediction processing unit 81, inverse quantization unit 86, inverse transform processing unit 88, adder 90, and DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra prediction unit 84, and an intra BC unit 85. Video decoder 30 may perform a decoding process that is substantially reciprocal to the encoding process described above in connection with fig. 2 with respect to video encoder 20. For example, the motion compensation unit 82 may generate prediction data based on the motion vector received from the entropy decoding unit 80, and the intra prediction unit 84 may generate prediction data based on the intra prediction mode indicator received from the entropy decoding unit 80.

In some examples, the elements of video decoder 30 may be tasked to perform embodiments of the present application. Further, in some examples, embodiments of the present disclosure may be dispersed in one or more of the units of video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in combination with other units of the video decoder 30 (e.g., the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80). In some examples, video decoder 30 may not include intra BC unit 85, and the functions of intra BC unit 85 may be performed by other components of prediction processing unit 81 (e.g., motion compensation unit 82).

Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. The video data stored in the video data memory 79 may be obtained, for example, from the storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). The video data memory 79 may include a Coded Picture Buffer (CPB) that stores coded video data from a coded video bitstream. DPB92 of video decoder 30 stores reference video data for use by video decoder 30 (e.g., in intra-prediction encoding mode or inter-prediction encoding mode) in decoding the video data. Video data memory 79 and DPB92 may be formed of any of a variety of memory devices, such as Dynamic Random Access Memory (DRAM), including Synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, video data memory 79 and DPB92 are depicted in fig. 3 as two different components of video decoder 30. It will be apparent to those skilled in the art that video data memory 79 and DPB92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip with other components of video decoder 30, or off-chip with respect to those components.

During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks of encoded video frames and associated syntax elements. Video decoder 30 may receive syntax elements at the video frame level and/or the video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors, or intra-prediction mode indicators, as well as other syntax elements. Entropy decoding unit 80 then forwards the motion vector or intra prediction mode indicator, as well as other syntax elements, to prediction processing unit 81.

When a video frame is encoded as an intra prediction encoded (I) frame or an intra encoding prediction block used in other types of frames, the intra prediction unit 84 of the prediction processing unit 81 may generate prediction data for a video block of the current video frame based on the signaled intra prediction mode and reference data from a previously decoded block of the current frame.

When a video frame is encoded as an inter-prediction encoded (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vectors and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks may be generated from reference frames within one of the reference frame lists. Video decoder 30 may construct a list of reference frames, i.e., list 0 and list 1, using a default construction technique based on the reference frames stored in DPB 92.

In some examples, when video blocks are encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block defined by video encoder 20.

The motion compensation unit 82 and/or the intra BC unit 85 determine prediction information for the video block of the current video frame by parsing the motion vector and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine a prediction mode (e.g., intra-prediction or inter-prediction) for encoding a video block of a video frame, an inter-prediction frame type (e.g., B or P), construction information for one or more of a reference frame list for the frame, a motion vector for each inter-prediction encoded video block of the frame, an inter-prediction state for each inter-prediction encoded video block of the frame, and other information for decoding the video block in the current video frame.

Similarly, the intra BC unit 85 may use some of the received syntax elements, such as flags, to determine that the current video block is construction information for some video blocks of the frame within the reconstruction region and that should be stored in the DPB 92 that are predicted using intra BC mode, block vectors for each intra BC predicted video block of the frame, intra BC prediction status for each intra BC predicted video block of the frame, and other information for decoding video blocks in the current video frame.

Motion compensation unit 82 may also perform interpolation using interpolation filters, such as those used by video encoder 20 during encoding of video blocks, to calculate interpolation values for sub-integer pixels of the reference block. In this case, motion compensation unit 82 may determine interpolation filters used by video encoder 20 from the received syntax elements and use these interpolation filters to generate the prediction block.

The inverse quantization unit 86 performs inverse quantization on quantized transform coefficients provided in the bit stream and entropy decoded by the entropy decoding unit 80 using the same quantization parameters calculated by the video encoder 20 for each video block in the video frame to determine the quantization degree. The inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to reconstruct the residual block in the pixel domain.

After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs a decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. Loop filter 91 (e.g., a deblocking filter, SAO filter, and/or ALF) may be located between adder 90 and DPB 92 to further process the decoded video blocks. In some examples, loop filter 91 may be omitted and the decoded video block may be provided directly to DPB 92 by adder 90. The decoded video blocks in a given frame are then stored in DPB 92, and DPB 92 stores reference frames for subsequent motion compensation of the next video block. DPB 92 or a memory device separate from DPB 92 may also store decoded video for later presentation on a display device (e.g., display device 34 of fig. 1).

In a typical video encoding process, a video sequence generally includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luminance samples. SCb is a two-dimensional array of Cb chroma-sampling points. SCr is a two-dimensional array of Cr chroma-sampling points. In other examples, the frame may be monochromatic, and thus include only one two-dimensional array of luminance samples.

As shown in fig. 4A, video encoder 20 (or more specifically, partitioning unit 45) generates an encoded representation of a frame by first partitioning the frame into a set of CTUs. The video frame may include an integer number of CTUs ordered consecutively from left to right and top to bottom in raster scan order. Each CTU is the largest logical coding unit and the width and height of the CTU are signaled by video encoder 20 in the sequence parameter set such that all CTUs in the video sequence have the same size of one of 128 x 128, 64 x 64, 32 x 32, and 16 x 16. It should be noted that the application is not necessarily limited to a particular size. As shown in fig. 4B, each CTU may include one CTB of a luminance sample, two corresponding coding tree blocks of a chrominance sample, and a syntax element for coding the samples of the coding tree blocks. Syntax elements describe the nature of the different types of units encoding the pixel blocks and how the video sequence may be reconstructed at video decoder 30, including inter-or intra-prediction, intra-prediction modes, motion vectors, and other parameters. In a monochrome picture or a picture having three separate color planes, a CTU may comprise a single coding tree block and syntax elements for encoding samples of the coding tree block. The coding tree block may be an nxn block of samples.

To achieve better performance, video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, trigeminal tree partitioning, quadtree partitioning, or a combination thereof, on the coded tree blocks of CTUs and divide the CTUs into smaller CUs. As depicted in fig. 4C, a 64 x 64 CTU 400 is first divided into four smaller CUs, each having a block size of 32 x 32. Among the four smaller CUs, the CUs 410 and 420 are divided into four CUs with block sizes of 16×16, respectively. Two 16×16 CUs 430 and 440 are each further divided into four CUs with block sizes of 8×8. Fig. 4D depicts a quadtree data structure showing the final result of the segmentation process of CTU 400 as depicted in fig. 4C, each leaf node of the quadtree corresponding to one CU of a respective size ranging from 32 x 32 to 8 x 8. Similar to the CTU depicted in fig. 4B, each CU may include two corresponding encoded blocks of CBs and chroma samples of luma samples of the same size frame, and syntax elements for encoding the samples of the encoded blocks. In a monochrome picture or a picture having three separate color planes, a CU may comprise a single coding block and syntax structures for encoding samples of the coding block. It should be noted that the quadtree partitions depicted in fig. 4C and 4D are for illustrative purposes only, and that one CTU may be split into multiple CUs based on quadtree partitions/trigeminal partitions/binary tree partitions to accommodate varying local characteristics. In a multi-type tree structure, one CTU is partitioned according to a quadtree structure, and each quadtree leaf CU may be further partitioned according to a binary and trigeminal tree structure. As shown in fig. 4E, there are five possible segmentation types for a coding block having a width W and a height H, namely, quaternary segmentation, horizontal binary segmentation, vertical binary segmentation, horizontal ternary segmentation, and vertical ternary segmentation.

In some implementations, video encoder 20 may further partition the coding block of the CU into one or more (mxn) PB. PB is a rectangular (square or non-square) block of samples to which the same prediction (inter or intra) is applied. The PU of a CU may include a PB of a luma sample, two corresponding PB of chroma samples, and syntax elements for predicting the PB. In a monochrome picture or a picture having three separate color planes, a PU may include a single PB and syntax structures for predicting the PB. Video encoder 20 may generate a predicted luma (Y) block, a predicted Cb block, and a predicted Cr block for luma PB, cb PB, and Cr PB of each PU of the CU.

Video encoder 20 may use intra-prediction or inter-prediction to generate a prediction block for the PU. If video encoder 20 uses intra-prediction to generate the prediction block of the PU, video encoder 20 may generate the prediction block of the PU based on decoded samples of the frame associated with the PU. If video encoder 20 uses inter prediction to generate the prediction block of the PU, video encoder 20 may generate the prediction block of the PU based on decoded samples of one or more frames other than the frame associated with the PU.

After video encoder 20 generates the predicted luma block, the predicted Cb block, and the predicted Cr block for the one or more PUs of the CU, video encoder 20 may generate a luma residual block for the CU by subtracting the predicted luma block of the CU from the original luma coded block of the CU such that each sample in the luma residual block of the CU indicates a difference between a luma sample in one of the predicted luma blocks of the CU and a corresponding sample in the original luma coded block of the CU. Similarly, video encoder 20 may generate Cb residual blocks and Cr residual blocks for the CU, respectively, such that each sample in the Cb residual block of the CU indicates a difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb encoded block of the CU, and each sample in the Cr residual block of the CU may indicate a difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr encoded block of the CU.

Further, as shown in fig. 4C, video encoder 20 may use quadtree partitioning to decompose a luma residual block, a Cb residual block, and a Cr residual block of the CU into one or more luma transform blocks, cb transform blocks, and Cr transform blocks, respectively. The transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. The TUs of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements for transforming the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of a luma residual block of a CU. The Cb transform block may be a sub-block of a Cb residual block of the CU. The Cr transform block may be a sub-block of a Cr residual block of the CU. In a monochrome picture or a picture having three separate color planes, a TU may comprise a single transform block and syntax structures for transforming the samples of the transform block.

Video encoder 20 may apply one or more transforms to the luma transform block of the TU to generate a luma coefficient block for the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar quantities. Video encoder 20 may apply one or more transforms to the Cb transform block of the TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to the Cr transform blocks of the TUs to generate Cr coefficient blocks for the TUs.

After generating the coefficient block (e.g., the luma coefficient block, the Cb coefficient block, or the Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to the process by which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements that indicate the quantized transform coefficients. For example, video encoder 20 may perform CABAC on syntax elements that indicate quantized transform coefficients. Finally, video encoder 20 may output a bitstream including a sequence of bits that form a representation of the encoded frames and associated data, which is stored in storage device 32 or transmitted to target device 14.

Upon receiving the bitstream generated by video encoder 20, video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct the frames of video data based at least in part on syntax elements obtained from the bitstream. The process of reconstructing video data is typically reciprocal to the encoding process performed by video encoder 20. For example, video decoder 30 may perform an inverse transform on the coefficient blocks associated with the TUs of the current CU to reconstruct residual blocks associated with the TUs of the current CU. Video decoder 30 also reconstructs the encoded block of the current CU by adding the samples of the prediction block for the PU of the current CU to the corresponding samples of the transform block of the TU of the current CU. After reconstructing the encoded blocks for each CU of the frame, video decoder 30 may reconstruct the frame.

As described above, video coding mainly uses two modes, i.e., intra-frame prediction (or intra-frame prediction) and inter-frame prediction (or inter-frame prediction), to achieve video compression. Note that IBC may be considered as intra prediction or third mode. Between the two modes, since motion vectors are used to predict the current video block from the reference video block, inter prediction contributes more to coding efficiency than intra prediction.

But with ever-improving video data capture techniques and finer video block sizes for preserving details in video data, the amount of data required to represent the motion vector for the current frame has also increased substantially. One way to overcome this challenge benefits from the fact that: not only are a set of neighboring CUs in both the spatial and temporal domains having similar video data for prediction purposes, but the motion vectors between these neighboring CUs are also similar. Thus, the motion information of a spatially neighboring CU and/or a temporally co-located CU may be used as an approximation of the motion information (e.g., motion vector) of the current CU (which is also referred to as "motion vector predictor" (MVP)) by: exploring their spatial and temporal correlation.

Instead of encoding the actual motion vector of the current CU, as determined by the motion estimation unit 42, into the video bitstream as described above in connection with fig. 2, the motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to generate a Motion Vector Difference (MVD) for the current CU. By doing so, it is not necessary to encode the motion vector determined by the motion estimation unit 42 for each CU of a frame into the video bitstream, and the amount of data representing the motion information in the video bitstream can be significantly reduced.

Similar to the process of selecting a prediction block in a reference frame during inter prediction of an encoded block, both video encoder 20 and video decoder 30 need to employ a set of rules for constructing a motion vector candidate list (also referred to as a "merge list") for the current CU using those potential candidate motion vectors associated with spatially neighboring CUs and/or temporally co-located CUs of the current CU, and then select one member from the motion vector candidate list as a motion vector predictor for the current CU. By doing so, there is no need to send the motion vector candidate list itself from video encoder 20 to video decoder 30, and the index of the selected motion vector predictor within the motion vector candidate list is sufficient for video encoder 20 and video decoder 30 to use the same motion vector predictor within the motion vector candidate list to encode and decode the current CU.

In VVC, an adaptive loop filter (ALF, adaptive loop filter) with block-based filter adaptation is applied. For the luminance component, one of 25 filters is selected for each 4 x 4 block based on the direction and activity of the local gradient.

In some embodiments, to reduce cross-component redundancy, a cross-component linear model (CCLM, cross-component linear model) prediction mode is used. Therefore, chroma samples are predicted based on reconstructed luma samples of the same CU by using the linear model as follows:

pred _C (i,j)＝α·rec _L '(i,j)+β

wherein pred _C (i, j) denotes the predicted chroma samples in the CU, rec _L 'j' denotes the downsampled reconstructed luma samples of the same CU. The linear model parameters α and β are derived from a linear relationship between luminance and chrominance values from two samples, which are the smallest luminance sample a (X _A ,Y _A ) And maximum luminance sample point B (X _B ,Y _B ). Where X is _A And Y _A Is for the X coordinate value (i.e. luminance value) and y coordinate value (i.e. chrominance value) of sample point a, X _B And Y _B Is the x and y coordinate values for sample point B. The linear model parameters α and β are obtained according to the following equation.

β＝y _A -αx _A

This approach is also referred to as the min-max approach as shown in fig. 5. Instead of using division in the above equation, multiplication and shifting may be used.

For a coded block having a square shape, the two equations above are directly applied. For non-square coded blocks, adjacent samples of longer boundaries are first sub-sampled to have the same number of samples as the shorter boundaries.

Fig. 6 illustrates the positions of the left-hand sample point and the current block sample point of the current block involved in the CCLM mode.

In some embodiments, in addition to the scenario in which the upper and left templates are used to calculate the linear model coefficients, these two templates may also be used interchangeably in the other two LM modes, referred to as lm_a and lm_l modes.

In lm_a mode, only pixel samples in the upper template are used to calculate linear model coefficients. To obtain more spots, the upper template is expanded to the size of (w+w). In lm_l mode, only pixel samples in the left template are used to calculate linear model coefficients. To get more samples, the left template is expanded to the size of (H+H).

In some examples, when the upper reference line is located at the CTU boundary, only one luma line (which is stored in a line buffer for intra prediction) is used to generate the downsampled luma samples.

For chroma intra mode codec, a total of 8 intra modes are allowed for chroma intra mode codec. These modes include five traditional intra modes and three cross-component linear model modes (CCLM, lm_a, and lm_l). The chroma mode signaling and derivation procedure is shown in table 1. Chroma mode codec depends directly on the intra prediction mode of the corresponding luma block. Since the individual block division structure for the luminance component and the chrominance component is enabled in the I slice, one chrominance block may correspond to a plurality of luminance blocks. Thus, for the chroma DM mode, it is directly inherited that the intra prediction mode of the corresponding luma block covers the center position of the current chroma block.

Table 1 deriving chroma prediction mode from luma mode when CCLM is enabled

Fig. 7 is a block diagram illustrating exemplary derivation of parameters for a multi-model linear model (MMLM, multi-model linear model) by classifying neighboring samples into two groups, according to some embodiments of the present disclosure.

In some embodiments, to reduce cross-component redundancy, a multi-mode linear model (MMLM) prediction mode is implemented. Chroma samples are predicted based on reconstructed luma samples of the same CU by using two linear models:

Wherein pred _C (i, j) denotes the predicted chroma samples in the CU, rec _L 'j' denotes the downsampled reconstructed luma samples of the same CU. Threshold is calculated as the average of neighboring reconstructed luminance samples. Fig. 7 shows an example of classifying neighboring samples into two groups based on a value Threshold. In this example, threshold is 17. For each group, the parameter α _i And beta _i (i is equal to 1 and 2, respectively) derived from a linear relationship between luminance and chrominance values from two samples, which are the smallest luminance samples A (X) _A ,Y _A ) And maximum luminance sample point B (X _B ,Y _B ). Where X is _A And Y _A Is for the X coordinate value (i.e. luminance value) and y coordinate value (i.e. chrominance value) of sample point a, X _B And Y _B Is the x and y coordinate values for sample point B. The linear model parameters α and β are obtained according to the following equation:

β＝y _A -αx _A

this approach is also known as the min-max approach. Instead of using division in the above equation, multiplication and shifting may be used.

In addition to the scenario where the upper and left templates are used together to calculate the linear model coefficients, these two templates can also be used interchangeably in the other two MMLM modes, referred to as mmlm_a and mmlm_l modes.

In mmlm_a mode, only pixel samples in the upper template are used to calculate linear model coefficients. To obtain more spots, the upper template is expanded to the size of (w+w). In mmlm_l mode, only pixel samples in the left template are used to calculate linear model coefficients. To get more samples, the left template is expanded to the size of (H+H).

For chroma intra mode codec, a total of 11 intra modes are allowed for chroma intra mode codec. These modes include five traditional intra modes and six cross-component linear model modes (CCLM, lm_ A, LM _ L, MMLM, MMLM _a, and mmlm_l). The chroma mode signaling and derivation procedure is shown in table 2. Chroma mode codec depends directly on the intra prediction mode of the corresponding luma block. Since the individual block division structure for the luminance component and the chrominance component is enabled in the I slice, one chrominance block may correspond to a plurality of luminance blocks. Thus, for the chroma DM mode, it is directly inherited that the intra prediction mode of the corresponding luma block covers the center position of the current chroma block.

Table 2 deriving chroma prediction mode from luma mode when MMLM is enabled

In some embodiments, adaptive enablement for predicted LMs and MMLMs is implemented. The MMLM and LM modes can also be used together in an adaptive manner. For MMLM, two linear models are as follows:

wherein pred _C (i, j) denotes the predicted chroma samples in the CU, rec _L 'j' denotes the downsampled reconstructed luma samples of the same CU. Threshold can be simply determined based on the luminance and chrominance averages, together with the luminance and chrominance minimums and maximums. Fig. 8 shows an example of classifying adjacent samples into two groups based on an inflection point T (indicated by an arrow). Linear model parameter alpha ₁ And beta ₁ Derived from a linear relationship between luminance and chrominance values from two samples, the minimum luminance sample a (X _A ,Y _A ) And Threshold (X) _T ,Y _T ). Linear model parameter alpha ₂ And beta ₂ Derived from a linear relationship between luminance and chrominance values from two samples, the maximum luminance sample B (X _B ,Y _B ) And Threshold (X) _T ,Y _T ). Where X is _A And Y _A Is for the X coordinate value (i.e. luminance value) and y coordinate value (i.e. chrominance value) of sample point a, X _B And Y _B Is the x and y coordinate values for sample point B. The linear model parameters alpha for each group _i And beta _i (i is equal to 1 and 2, respectively) is obtained according to the following equation:

β ₁ ＝Y _A -α ₁ X _A

β ₂ ＝Y _T -α ₂ X _T

for a coded block having a square shape, the above equation is directly applied. For non-square coded blocks, adjacent samples of longer boundaries are first sub-sampled to have the same number of samples as the shorter boundaries.

In addition to the scenario where the upper and left templates are used together to determine the coefficients of the linear model, these two templates can also be used interchangeably in the other two MMLM modes, referred to as mmlm_a and mmlm_l modes, respectively.

For chroma intra mode codec, there is one condition check to select either LM mode (CCLM, lm_a, and lm_l) or multi-mode LM mode (MMLM, mmlm_a, and mmam_l). The condition check is as follows:

wherein BlkSizeThres _LM Representing the minimum block size of the LM mode, blkSizeThres _MM Representing the minimum block size of the MMLM pattern. The symbol d represents a predetermined threshold. In one example, d may take on a value of 0. In another example, d may take on a value of 8.

For chroma intra mode codec, a total of 8 intra modes are allowed for chroma intra mode codec. These modes include five traditional intra modes and three cross-component linear model modes. The chroma mode signaling and derivation procedure is shown in table 3. Notably, for a given CU, if the CU is encoded in a linear model mode, it is determined whether the linear model mode is a conventional single model LM mode or MMLM mode based on the above condition check. Unlike the case shown in table 2, there is no separate MMLM mode to be signaled. Chroma mode codec depends directly on the intra prediction mode of the corresponding luma block. Since the individual block division structure for the luminance component and the chrominance component is enabled in the I slice, one chrominance block may correspond to a plurality of luminance blocks. Thus, for the chroma DM mode, it is directly inherited that the intra prediction mode of the corresponding luma block covers the center position of the current chroma block.

Table 3 deriving chroma prediction mode from luma mode when CCLM is enabled

In some embodiments, local illumination compensation (LIC, local Illumination Compensation) is an inter-prediction technique for modeling local illumination changes between a current block and its predicted block as a function of local illumination changes between the current block template and a reference block template. The parameters of the function can be represented by a scaling factor α and an offset β, so as to form a linear equation, i.e., α x + β, to compensate for the illumination variation, where p x is the reference sample point pointed by the MV at position x on the reference picture. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required for α and β other than signaling an LIC flag to indicate use of LIC for AMVP mode.

Fig. 9A and 9B are block diagrams illustrating exemplary neighboring samples for deriving illumination compensation (IC, illumination Compensation) parameters according to some embodiments of the present disclosure.

When LIC is applied to a CU, the parameters α and β are derived using a least squares error method by using neighboring samples of the current CU and their corresponding reference samples. More specifically, as shown in fig. 9A and 9B, the samples used are sub-sampled (2:1 sub-sampled) neighboring samples of the CU and corresponding samples in the reference picture (identified by the motion information of the current CU or sub-CU). The IC parameters are derived and applied to each prediction direction individually.

When LIC is enabled for pictures, an additional CU level RD check is needed to determine if LIC is applied for the CU. When LIC is enabled for CU, sum of mean-absolute differences (MR-SAD, mean-removed sum of absolute difference) and sum of mean-absolute hadamard transform differences (MR-SATD, mean-removed sum of absolute Hadamard-transformed difference) are used instead of SAD and SATD for integer image element motion search and fractional image element motion search, respectively.

In order to reduce the coding complexity, the following coding scheme is applied in JEM. When there is no significant change in illumination between the current picture and its reference picture, the LIC is disabled for the entire picture. To identify this, a histogram of the current picture is calculated at the encoder along with a histogram of each reference picture of the current picture. Disabling the LIC for the current picture if the histogram difference between the current picture and each reference picture of the current picture is less than a given threshold; otherwise, LIC is enabled for the current picture.

Although LIC can effectively model local illumination variations, its performance can still be improved. In addition, LIC designs also introduce significant complexity into encoder and decoder designs. Further optimization is needed to achieve the trade-off between complexity and codec efficiency.

In some embodiments of the present disclosure, several methods are implemented to further improve LIC codec efficiency or to simplify existing LIC designs to facilitate hardware implementation. The methods and systems disclosed herein may be applied independently or in conjunction with other systems or methods.

In some embodiments, to simplify the computational complexity of the LIC, generating model parameters for the LIC with a more limited number of reference samples is implemented to reduce the necessary computations.

In some embodiments, the number of reference samples used to derive the LIC parameters may be reduced (as compared to the methods currently employed in LIC) or limited to a range of specific values. In some embodiments, the calculation of Illumination Compensation (IC) parameters using a more limited number of reference samples is described further below. In some examples, model parameters for the LIC are generated with a more limited number of reference points to reduce the necessary computations. In one embodiment, only half of those reference sample point pairs currently in use are applied to determine LIC parameters in accordance with the present disclosure. (reference sample pair refers to neighboring luma samples of the current CU and their corresponding reference luma samples, e.g., neighboring reconstructed luma samples of the current block and corresponding neighboring reconstructed luma samples of the reference block). For example, by considering one of every two adjacent pairs of reference samples when deriving the IC model parameters, these reference samples may be selected in a spatially further downsampled manner.

In another embodiment, the maximum number of reference sample pairs used to derive the LIC model parameters is limited to a predetermined value based on the size and shape of the corresponding luminance block. Four different examples (labeled as methods 1, 2, 3, and 4) are provided in table 4, where the predetermined values may be 2, 4, and/or 8, depending on the size and shape of the luma block of the current CU (e.g., the size and shape of the luma block of the current block or the reference block).

Luminance block size	JEM	Method 1	Method 2	Method 3	Method 4
						Upper limit of reference point pair	64	8	8	4	4
2×n/n×2	4	2	4	4	2
						4×n/n×4(n≥4)	8	4	8	4	4
8×n/n×8(n≥8)	16	8	8	4	4
						16×n/n×16(n≥16)	32	8	8	4	4
32×32	64	8	8	4	4

Table 4 number of pairs of samples calculated for LIC parameters in JEM and method for implementing same

In a third embodiment of the present disclosure, only blocks with block sizes equal to or greater than a certain threshold may be used to form inter prediction of LIC. In one example, the minimum block size is limited to 8 or 16. In this case, the maximum number of reference sample pairs may be limited to 8.

In some embodiments, to improve codec efficiency, an adaptive LIC scheme is implemented. In contrast to the method where LIC is fixedly applied to one linear model, the algorithms and systems disclosed herein adaptively adjust the number of linear models.

In some embodiments, the multi-model local illumination compensation method is described further below. In a fourth embodiment of the present disclosure, one or more multi-model LIC (MMLIC) modes are added. In each MMLIC mode, reconstructed neighboring samples are classified into two classes using a threshold, which is the average of luminance reconstructed neighboring samples. A Least-Mean-Square (LMS) method is used to derive the linear model for each class.

For example, by using two linear models, a model for predicting a sample based on a reference sample pointed to by MV at position x on a reference picture is as follows:

where pred (i, j) denotes the prediction samples, rec' (i, j) denotes the reference samples pointed to by MV at position x on the reference picture. Threshold is calculated as the average of neighboring reconstructed samples. Fig. 7 shows an example of classifying neighboring samples into two groups based on a value Threshold. For each group, the parameter α _i And beta _i (i is equal to 1 and 2, respectively) derived from a linear relationship between adjacent luminance sample values from the current CU from two sample pairs, which are the smallest luminance sample points a (X _A ,Y _A ) And maximum luminance sample point B (X _B ,Y _B ). Where X is _A And Y _A Is for the X-coordinate value (i.e. the luminance value) and the y-coordinate value (i.e. the corresponding reference luminance value) of sample point a, X _B And Y _B Is the x and y coordinate values for sample point B. The linear model parameters α and β are obtained according to the following equation:

β＝y _A -αx _A

Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required for α and β other than signaling the use of MMLIC flags to indicate MMLIC.

In addition to the scenario where the upper and left templates are used together to calculate the linear model coefficients, these two templates can also be used interchangeably in the other two MMLIC modes, referred to as MMLIC_A mode and MMLIC_L mode.

In the MMLIC_A mode, only the pixel samples in the upper template are used to calculate the linear model coefficients. To obtain more spots, the upper template is expanded to the size of (w+w). In the MMLIC_L mode, only the pixel samples in the left template are used to calculate the linear model coefficients. To get more samples, the left template is expanded to the size of (H+H).

In a fifth embodiment of the present disclosure, parameter derivation in LIC may use MMLIC mode only. For example, only one or more MMLIC modes are allowed and single model based LIC modes are disabled. In this case, it is not necessary to determine whether to apply the LIC mode or the multimode LIC mode. For example, a condition check for selecting an LIC mode or a multi-model LIC mode is no longer required. The multi-model LIC mode is always used for parameter derivation and sample value prediction in LIC.

In some embodiments, block-based pixel classification for model selection in MMLIC mode is described below. In a sixth embodiment of the present disclosure, block-based pixel classification is used to select different models in MMLIC mode. Currently, such classification is pixel-based, i.e. when a corresponding LIC model is selected for that pixel, compared to a classification threshold and each reconstructed luminance sample is examined based on the comparison result. According to this embodiment of the present disclosure, this classification is done at the block level, where the classification decision applies to all pixels in the block. In one example, the block size may be n×m, where N and M are positive numbers such as 2 or 4. Taking the example that both N and M are equal to 2, the classification in this case is done on a 2 x 2 block level. As a result, the same linear model will be selected for all four pixels in the block.

In accordance with the present disclosure, different methods may be used for classification, including all or only a portion of the samples in a block. For example, the average of all samples in each nxm block may be used to determine which linear model to use for the block. In another example, for simplicity, classification may be performed by simply examining one sample from each block to determine which linear model to use for that block. The one sample may be the left hand sample of each nxm block.

In some embodiments, a three-model based local illumination compensation is implemented. In a seventh embodiment of the present disclosure, three parameter sets are used in the local illumination compensation mode to compensate for illumination variation. In one embodiment, the parameters of the function may be represented by a scaling factor α and an offset β, forming a linear equation that predicts luminance samples based on reconstructed luminance samples of the same CU in the reference picture (or reference block) by using three linear models: alpha x + beta

Wherein pred (i, j) represents the predicted luminance samples, rec' _L (i, j) represents the reference sample pointed to by the MV at position (i, j) on the reference picture. In one embodiment, threshold ₁ And Threshold ₂ Can be calculated by adjacent maximum and minimum values (hereinafter denoted Lmax and Lmin, respectively) of reconstructed luminance samples. In one example, threshold ₁ And Threshold ₂ The calculation can be as follows:

in an eighth embodiment of the present disclosure, threshold ₁ And Threshold ₂ May be calculated as an average of neighboring reconstructed luma samples. In one example, all neighboring reconstructed luma samples are divided into two groups based on an average value of the neighboring reconstructed luma samples. Luminance samples having a value smaller than the average value belong to a group, have a value not smaller than the average valueLuminance samples of the value of the mean belong to another group. Threshold ₁ And Threshold ₂ The average value for each group can be calculated. At the time of determining Threshold ₁ And Threshold ₂ Can depend on the luminance value and Threshold after the value of (2) ₁ And Threshold ₂ The relation between the values of (a) divides adjacent reconstructed luma samples into three groups. For example, the first group includes a first group having a value ranging from a minimum luminance sample value to Threshold ₁ Is used to reconstruct the luminance samples. The second group comprises a first group having a value ranging from Threshold ₁ To Threshold ₂ Is used to reconstruct the luminance samples. The third group contains the remaining reconstructed luma samples.

With the sample points divided into three groups, linear model parameters can be derived for each group. In one example, the parameters α and β are derived from a linear relationship between neighboring luminance sample values from the current CU from two sample pairs, which are the minimum and maximum luminance samples within each of the three groups, and their corresponding reference luminance sample values, respectively. In another example, the linear model parameter α ₁ And beta ₁ Derived from a linear relationship between adjacent luma sample values of the current CU from two sample pairs, the minimum luma sample and Threshold, and their corresponding reference luma sample values ₁ . Linear model parameter alpha ₂ And beta ₂ Derived from a linear relationship between adjacent luma sample values of the current CU from two sample pairs, which are Threshold, and their corresponding reference luma sample values ₁ And Threshold ₂ . Linear model parameter alpha ₃ And beta ₃ Derived from a linear relationship between adjacent luminance sample values of the current CU from two sample pairs, the maximum luminance sample and Threshold, and their corresponding reference luminance sample values ₂ 。

In some embodiments, the model classification threshold is determined based on reconstructed luma samples within the current CU. In a ninth embodiment of the present disclosure, reconstructed luma samples within the current CU are used to calculate model classification thresholds in a cross-component linear model. In one embodiment, the threshold is obtained by calculating an average of reconstructed luma samples within the CU. In another embodiment, the threshold is obtained by calculating an average of reconstructed luma samples within the CU and reconstructed luma samples adjacent to the CU.

In some embodiments, the model classification threshold is based on a minimum luminance sample and a maximum luminance sample. In a tenth embodiment of the present disclosure, the minimum and maximum samples are used to derive model classification thresholds. In one embodiment, the threshold is calculated as (max+min)/N, where max is the sample value of the largest sample, min is the sample value of the smallest sample, and N is any value (e.g., 2).

Fig. 14 is a flowchart illustrating an exemplary process 1400 of decoding a video signal according to some embodiments of the present disclosure.

Video decoder 30 (shown in fig. 3) determines two or more reference sample pairs, each reference sample pair comprising an adjacent reconstructed luma sample of the current block and a corresponding adjacent reconstructed luma sample of the reference block (1410).

Video decoder 30 classifies the two or more reference sample pairs into one or more groups (1420).

Video decoder 30 derives one or more linear models based on the classified one or more group sample pairs (1430).

Video decoder 30 predicts luminance sample values in the current block by applying one or more linear models to the corresponding reconstructed luminance samples in the reference block (1440).

In some embodiments, video decoder 30 adjusts the luma sample values in the current block based on the predicted luma sample values.

In some embodiments, the reference block is derived from the current block by a motion vector shift.

In some embodiments, determining two or more reference sample pairs (1410) includes: the predetermined number of sample pairs is determined based on the size and shape of the luminance block of the reference block or the current block.

In some embodiments, the predetermined number of pairs of samples is 2, 4, or 8.

In some embodiments, predicting luma sample values in a current block (1440) includes: luminance sample values in the current block are predicted by applying one or more linear models to corresponding reconstructed luminance samples in a reference block having a block size equal to or greater than a threshold.

In some embodiments, deriving one or more linear models (1430) based on the classified one or more group-sample pairs includes: two or more linear models are derived based on the classified two or more sets of pairs of samples, and each linear model corresponds to each set of pairs of samples.

In some embodiments, classifying the two or more reference sample pairs into one or more groups (1420) includes: two or more reference sample pairs are classified into one or more groups according to the values associated with the pixels in the reference block.

In some embodiments, the value associated with a pixel in the reference block is an average of all pixels in the reference block or a sample value selected from all pixels in the reference block.

In some embodiments, deriving one or more linear models (1430) based on the classified one or more group-sample pairs includes: three linear models are derived based on the three sets of sample pairs obtained by the classification, and each linear model corresponds to each set of sample pairs.

In some embodiments, classifying two or more reference sample pairs into one or more groups (1420) includes: two or more reference sample pairs are classified into one or more groups based on one or more thresholds, and the one or more thresholds are based on reconstructed luma samples within a reference block and/or an average of neighboring reconstructed luma samples of the reference block.

In some embodiments, classifying two or more reference sample pairs into one or more groups (1420) includes: two or more reference sample pairs are classified into one or more groups based on one or more thresholds, and the one or more thresholds are calculated from (max+min)/N, where MAX is a maximum sample value of reconstructed luminance samples within the reference block, MIN is a minimum sample value of reconstructed luminance samples within the reference block, and N is a predetermined value.

Fig. 15 illustrates a computing environment 1510 coupled to a user interface 1550. The computing environment 1510 may be part of a data processing server. The computing environment 1510 includes a processor 1520, memory 1530, and input/output (I/O) interfaces 1540.

The processor 1520 generally controls overall operation of the computing environment 1510, such as operations associated with display, data acquisition, data communication, and image processing. Processor 1520 may include one or more processors for executing instructions to perform all or some of the steps of the methods described above. Further, the processor 1520 may include one or more modules that facilitate interactions between the processor 1520 and other components. The processor may be a Central Processing Unit (CPU), microprocessor, single-chip microcomputer, graphics Processing Unit (GPU), or the like.

The memory 1530 is configured to store various types of data to support the operation of the computing environment 1510. Memory 1530 may include predetermined software 1532. Examples of such data include instructions, video data sets, image data, and the like for any application or method operating on the computing environment 1510. The memory 1530 may be implemented using any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically Erasable Programmable Read Only Memory (EEPROM), erasable Programmable Read Only Memory (EPROM), programmable Read Only Memory (PROM), read Only Memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

The I/O interface 1540 provides an interface between the processor 1520 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I/O interface 1540 may be coupled with an encoder and a decoder.

In an embodiment, there is also provided a non-transitory computer readable storage medium comprising, for example, a plurality of programs in the memory 1530 executable by the processor 1520 in the computing environment 1510 for performing the above-described methods. Alternatively, a non-transitory computer readable storage medium may have stored therein a bitstream or data stream comprising encoded video information (e.g., video information comprising one or more syntax elements) that is generated by an encoder (e.g., video encoder 20 of fig. 2) using, for example, the encoding method described above, for use by a decoder (e.g., video decoder 30 of fig. 3) in decoding video data. The non-transitory computer readable storage medium may be, for example, ROM, random-access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

In an embodiment, there is also provided a computing device including: one or more processors (e.g., processor 1520); and a non-transitory computer readable storage medium or memory 1530 having stored therein a plurality of programs executable by one or more processors, wherein the one or more processors are configured to perform the above-described methods when the plurality of programs are executed.

In an embodiment, there is also provided a computer program product comprising a plurality of programs, e.g., in memory 1530, executable by processor 1520 in computing environment 1510 for performing the methods described above. For example, the computer program product may include a non-transitory computer readable storage medium.

In an embodiment, the computing environment 1510 may be implemented by one or more ASICs, DSPs, digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the methods described above.

Other embodiments also include various subsets of the above embodiments combined or otherwise rearranged in various other embodiments.

In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium, and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media, such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another (e.g., according to a communication protocol). In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures to implement the embodiments described herein. The computer program product may include a computer-readable medium.

The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosure. Many modifications, variations and alternative embodiments will come to mind to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.

The order of steps of the method according to the present disclosure is intended to be illustrative only, unless specifically stated otherwise, and the steps of the method according to the present disclosure are not limited to the above-described order, but may be changed according to actual circumstances. Furthermore, at least one of the steps of the method according to the present disclosure may be adjusted, combined or pruned as actually needed.

The examples were chosen and described in order to explain the principles of the present disclosure and to enable others skilled in the art to understand the disclosure for various embodiments and with various modifications as are suited to the particular use contemplated. Therefore, it is to be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and that modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A method of decoding a video signal, the method comprising:

determining two or more reference sample point pairs, each of the reference sample point pairs comprising an adjacent reconstructed luma sample point of the current block and a corresponding adjacent reconstructed luma sample point of the reference block;

classifying the two or more reference sample point pairs into one or more groups;

deriving one or more linear models based on the classified one or more group sample point pairs; and

luminance sample values in the current block are predicted by applying the one or more linear models to corresponding reconstructed luminance samples in the reference block.

2. The method of claim 1, wherein the reference block is derived from the current block by a motion vector shift.

3. The method of claim 1, wherein the determining two or more reference sample pairs comprises: a predetermined number of sample pairs is determined based on the size and shape of the luminance block of the reference block or the current block.

4. A method according to claim 3, wherein the predetermined number of pairs of spots is 2, 4 or 8.

5. The method of claim 1, wherein the predicting luma sample values in the current block comprises: the luminance sample values in the current block are predicted by applying the one or more linear models to the corresponding reconstructed luminance samples in the reference block having a block size equal to or greater than a threshold.

6. The method of claim 1, wherein the deriving one or more linear models based on the classified one or more group-like point pairs comprises: two or more linear models are derived based on the classified two or more sets of pairs of samples, and each linear model corresponds to each set of pairs of samples.

7. The method of claim 1, wherein the classifying the two or more reference sample pairs into one or more groups comprises: the two or more reference sample pairs are classified into one or more groups according to values associated with pixels in the reference block.

8. The method of claim 7, wherein the value associated with a pixel in the reference block is an average of all pixels in the reference block or a sample value selected from all pixels in the reference block.

9. The method of claim 1, wherein the deriving one or more linear models based on the classified one or more group-like point pairs comprises: three linear models are derived based on the three sets of sample pairs obtained by the classification, and each linear model corresponds to each set of sample pairs.

10. The method of claim 1, wherein the classifying the two or more reference sample pairs into one or more groups comprises: the two or more reference sample pairs are classified into one or more groups based on one or more thresholds, and the one or more thresholds are based on reconstructed luma samples within the reference block and/or an average of the neighboring reconstructed luma samples of the reference block.

11. The method of claim 1, wherein the classifying the two or more reference sample pairs into one or more groups comprises: the two or more reference sample pairs are classified into one or more groups based on one or more thresholds, and the one or more thresholds are calculated from (max+min)/N, where MAX is a maximum sample value of the reconstructed luminance samples within the reference block, MIN is a minimum sample value of the reconstructed luminance samples within the reference block, and N is a predetermined value.

12. An electronic device, comprising:

one or more processing units;

a memory coupled to the one or more processing units; and

a plurality of programs stored in the memory, which when executed by the one or more processing units, cause the electronic device to perform the method of any of claims 1-11.

13. A computer readable storage medium storing a bitstream comprising video information generated by the method of decoding a video signal according to any of claims 1-11.

14. A non-transitory computer readable storage medium storing a plurality of programs for execution by an electronic device with one or more processing units, wherein the plurality of programs, when executed by the one or more processing units, cause the electronic device to perform the method of any of claims 1-11.