EP4677853A1 - A method, an apparatus and a computer program product for video encoding and decoding - Google Patents

Abstract

The present embodiments relate to a method and technical equipment for implementing the method. The method comprises processing (1010) a video frame to predict a block according to prediction samples; determining (1020) when both prediction blocks (910, 920) for a block are within boundaries of an image, and applying a bi-prediction based on said prediction blocks for the block resulting in a bi-prediction block (952); determining (1030) when one of the prediction blocks is at least partially out of image's boundaries, and applying a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block (951); deriving (1040) filter coefficients by minimizing squared error between a bi-prediction block and a uni-prediction block; filtering (1050) at least a part of the uni-prediction block (951) according to the filter coefficients to generate a filtered uni-prediction block. The embodiments also concern a technical equipment for implementing the method.

Description

A METHOD, AN APPARATUS AND A COMPUTER PROGRAM PRODUCT FOR VIDEO ENCODING AND DECODING Technical Field [1] The present solution generally relates to video encoding and video decoding. In particular, the present solution relates to Background [2] This section is intended to provide a background or context to the invention that is recited in the claims. The description herein may include concepts that could be pursued but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section. [3] A video coding system may comprise an encoder that transforms an input video into a compressed representation suited for storage/transmission and a decoder that can uncompress the compressed video representation back into a viewable form. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form, for example, to enable the storage/transmission of the video information at a lower bitrate than otherwise might be needed. Summary [4] The scope of protection sought for various embodiments of the invention is set out by the independent claims. The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention. [5] Various aspects include a method, an apparatus and a computer readable medium comprising a computer program stored therein, which are characterized by what is stated in the independent claims. Various embodiments are disclosed in the dependent claims. [6] According to a first aspect, there is provided an apparatus comprising means for means for processing a video frame to predict a block according to prediction samples; means for determining when both prediction blocks for a block are within boundaries of an image, and means for applying a bi- prediction based on said prediction blocks for the block resulting in a bi- prediction block; means for determining when one of the prediction blocks is at least partially out of image’s boundaries, and means for applying a uni- prediction based on the prediction block within image boundaries resulting in a uni-prediction block; means for deriving filter coefficients by minimizing squared error between the bi-prediction block and the uni-prediction block; and means for filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block. [7] According to a second aspect, there is provided a method, comprising: processing a video frame to predict a block according to prediction samples; determining when both prediction blocks for a block are within boundaries of an image, and applying a bi-prediction based on said prediction blocks for the block resulting in a bi-prediction block; determining when one of the prediction blocks is at least partially out of image’s boundaries, and applying a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; deriving filter coefficients by minimizing squared error between the bi-prediction block and the uni- prediction block; and filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block. [8] According to a third aspect, there is provided an apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: process a video frame to predict a block according to prediction samples; determine when both prediction blocks for a block are within boundaries of an image, and apply a bi-prediction based on said prediction blocks for the block resulting in a bi-prediction block; determin when one of the prediction blocks is at least partially out of image’s boundaries, and apply a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; derive filter coefficients by minimizing squared error between the bi-prediction block and the uni-prediction block; and filter at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block. [9] According to a fourth aspect, there is provided computer program product comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to: process a video frame to predict a block according to prediction samples; determine when both prediction blocks for a block are within boundaries of an image, and apply a bi-prediction based on said prediction blocks for the block resulting in a bi- prediction block; determin when one of the prediction blocks is at least partially out of image’s boundaries, and apply a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; derive filter coefficients by minimizing squared error between the bi-prediction block and the uni-prediction block; and filter at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block. [10] According to an embodiment, a use of filtering is indicated in samples of the uni-prediction block or bi-prediction block. [11] According to an embodiment, the filter contains non-linear terms. [12] According to an embodiment, the filter coefficients are derived for a set of samples that belong to an overlapping area of the bi-prediction block. [13] According to an embodiment, more than one filter is derived, wherein each filter uses its own area for filter derivation and is applied in its own area over the uni-prediction block. [14] According to an embodiment, an area is indicated on which the filter coefficients are derived in samples of the uni-predictive block or in the bi- prediction block. [15] According to an embodiment, an area of which the filter coefficients are derived is signaled at a coding unit, a prediction unit, a coding tree unit, a slice, a frame or a sequence level. [16] According to an embodiment, the computer program product is embodied on a non-transitory computer readable medium. Description of the Drawings [17] In the following, various embodiments will be described in more detail with reference to the appended drawings, in which [18] Fig.1 shows an example of the location of the left and above samples of a current block involved in the CCLM mode. [19] Fig.2a shows an example of derivation of chroma prediction mode from luma mode when CCLM is enabled. [20] Fig.2b shows an example of a unified binarization table for chroma prediction mode. [21] Fig.3a shows two luma-to-chrome models obtained for luma Y threshold of 17. [22] Fig.3b shows an example of correspondence of each luma-to- chroma model to a spatial segmentation of the content. [23] Fig.4 shows an example of locations of the samples used for the derivation of CCCM filter. [24] Fig.5 shows examples of various filter kernels; [25] Fig.6 shows an example of four reference lines neighboring to a prediction block. [26] Fig.7 shows an example an example of matrix weighted intra prediction process. [27] Fig.8 shows an example of top and left neighboring blocks used in CIIP weight derivation. [28] Fig.9 shows an example of a bi-predictive block; [29] Fig.10 shows an example of overlapping parts and non- overlapping parts; [30] Fig.11 shows an example of filter coefficients; [31] Fig.12 shows an example of two sets of filter coefficients; [32] Fig.13 is a flowchart illustrating a method according to an [33] Fig.14 shows an apparatus according to an embodiment. [34] Fig.15 shows an encoding process according to an embodiment. [35] Fig.16 shows a decoding process according to an embodiment. Embodiments [36] The following description and drawings are illustrative and are not to be construed as unnecessarily limiting. The specific details are provided for a thorough understanding of the disclosure. However, in certain instances, well- known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure can be, but not necessarily are, reference to the same embodiment and such references mean at least one of the embodiments. [37] Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. [38] In the following, several embodiments will be described in the context of one video coding arrangement. It is to be noted, however, that the present embodiments are not necessarily limited to this particular arrangement. The embodiments relate to error minimization based adjustment of bi-prediction in inter prediction. [39] The Advanced Video Coding standard (which may be abbreviated AVC or H.264/AVC) was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunications Standardization Sector of International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264/AVC standard is published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.264 and ISO/IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There have been multiple versions of the H.264/AVC standard, each integrating new extensions or features to the specification. These extensions include Scalable Video Coding (SVC) and Multiview Video Coding (MVC). [40] The High Efficiency Video Coding standard (which may be abbreviated HEVC or H.265/HEVC) was developed by the Joint Collaborative Team - Video Coding (JCT-VC) of VCEG and MPEG. The standard is published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.265 and ISO/IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Extensions to H.265/HEVC include scalable, multiview, three-dimensional, and fidelity range extensions, which may be referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively. The references in this description to H.265/HEVC, SHVC, MV-HEVC, 3D-HEVC and REXT that have been made for the purpose of understanding definitions, structures or concepts of these standard specifications are to be understood to be references to the latest versions of these standards that were available before the date of this application, unless otherwise indicated. [41] Versatile Video Coding (which may be abbreviated VVC, H.266, or H.266/VVC) is a video compression standard developed as the successor to HEVC. VVC is specified in ITU-T Recommendation H.266 and equivalently in ISO/IEC 23090-3, which is also referred to as MPEG-I Part 3. [42] A specification of the AV1 bitstream format and decoding process were developed by the Alliance of Open Media (AOM). The AV1 specification was published in 2018. AOM is reportedly working on the AV2 specification. [43] Some key definitions, bitstream and coding structures, and concepts of H.264/AVC, HEVC, VVC, and/or AV1 and some of their extensions are described in this section as an example of a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein the embodiments may be implemented. The aspects of various embodiments are not limited to H.264/AVC, HEVC, VVC, and/or AV1 or their extensions, but rather the description is given for one possible basis on top of which the present embodiments may be partly or fully realized. [44] A video codec may comprise an encoder that transforms the input video into a compressed representation suited for storage/transmission and a decoder that can uncompress the compressed video representation back into a viewable form. The compressed representation may be referred to as a bitstream or a video bitstream. A video encoder and/or a video decoder may also be separate from each other, i.e., need not form a codec. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate). The notation “(de)coder” means an encoder and/or a decoder. [45] In some video codecs, such as H.265/HEVC, video pictures are divided into coding units (CU) covering the area of the picture. A CU consists of one or more prediction units (PU) defining the prediction process for the samples within the CU and one or more transform units (TU) defining the prediction error coding process for the samples in the said CU. CU may consist of a square block of samples with a size selectable from a predefined set of possible CU sizes. A CU with the maximum allowed size may be named as LCU (largest coding unit) or CTU (coding tree unit), and the video picture is divided into non-overlapping CTUs. A CTU can be further split into a combination of smaller CUs, e.g., by recursively splitting the CTU and resultant CUs. Each resulting CU may have at least one PU and at least one TU associated with it. Each PU and TU can be further split into smaller PUs and TUs in order to increase granularity of the prediction and prediction error coding processes, respectively. Each PU has prediction information associated with it defining what kind of a prediction is to be applied for the pixels within that PU (e.g., motion vector information for inter predicted PUs and intra prediction directionality information for intra predicted PUs). Similarly, each TU is associated with information describing the prediction error decoding process for the samples within the said TU (including e.g., DCT coefficient information). It may be signaled at CU level whether prediction error coding is applied or not for each CU. In the case there is no prediction error residual associated with the CU, it can be considered there are no TUs for the said CU. The division of the image into CUs, and division of CUs into PUs and TUs may be signaled in the bitstream allowing the decoder to reproduce the intended structure of these units. [46] Hybrid video codecs, for example ITU-T H.263, H.264/AVC and HEVC, may encode the video information in two phases. At first, pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). In the first phase, predictive coding may be applied, for example, as so-called sample prediction and/or so-called syntax prediction. [47] In the sample prediction, pixel or sample values in a certain picture area or "block" are predicted. These pixel or sample values can be predicted, for example, using one or more of motion compensation or intra prediction mechanisms. [48] Motion compensation mechanisms (which may also be referred to as inter prediction, temporal prediction or motion-compensated temporal prediction or motion-compensated prediction or MCP) involve finding and indicating an area in one of the previously encoded video frames that corresponds closely to the block being coded. Inter prediction may reduce temporal redundancy. [49] Intra prediction, where pixel or sample values can be predicted by spatial mechanisms, involve finding and indicating a spatial region relationship. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied. [50] In the syntax prediction, which may also be referred to as parameter prediction, syntax elements and/or syntax element values and/or variables derived from syntax elements are predicted from syntax elements (de)coded earlier and/or variables derived earlier. Non-limiting examples of syntax prediction are provided below. [51] In motion vector prediction, motion vectors e.g., for inter and/or inter- view prediction may be coded differentially with respect to a block-specific predicted motion vector. In many video codecs, the predicted motion vectors are created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions, sometimes referred to as advanced motion vector prediction (AMVP), is to generate a list of candidate predictions from adjacent blocks and/or co-located blocks in temporal reference pictures and signalling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded/decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and/or co-located blocks in temporal reference picture. Differential coding of motion vectors is typically disabled across slice boundaries. [52] The block partitioning, e.g., from CTU to CUs and down to PUs, may be predicted. [53] In filter parameter prediction, the filtering parameters e.g., for sample adaptive offset may be predicted. [54] Prediction approaches using image information from a previously coded image can also be called as inter prediction methods which may also be referred to as temporal prediction and motion compensation. [55] Prediction approaches using image information within the same image can also be called as intra prediction methods. [56] Secondly, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This may be done by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size of transmission bitrate). [57] An elementary unit for the input to an encoder and the output of a decoder, respectively, in most cases is a picture. A picture given as an input to an encoder may also be referred to as a source picture, and a picture decoded by a decoded may be referred to as a decoded picture or a reconstructed picture. [58] The source and decoded pictures are each comprised of one or more sample arrays, such as one of the following sets of sample arrays: − Luma (Y) only (monochrome). − Luma and two chroma (YCbCr or YCgCo). − Green, Blue and Red (GBR, also known as RGB). − Arrays representing other unspecified monochrome or tri-stimulus color samplings (for example, YZX, also known as XYZ). [59] In the following, these arrays may be referred to as luma (or L or Y) and chroma, where the two chroma arrays may be referred to as Cb and Cr; regardless of the actual color representation method in use. The actual color representation method in use can be indicated e.g., in a coded bitstream e.g., using the Video Usability Information (VUI) syntax of HEVC or alike. A component may be defined as an array or single sample from one of the three sample arrays (luma and two chroma) or the array or a single sample of the array that compose a picture in monochrome format. [60] A picture may be defined to be either a frame or a field. A frame comprises a matrix of luma samples and possibly the corresponding chroma samples. A field is a set of alternate sample rows of a frame and may be used as encoder input, when the source signal is interlaced. Chroma sample arrays may be absent (and hence monochrome sampling may be in use) or chroma sample arrays may be subsampled when compared to luma sample arrays. [61] The decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain). After applying prediction and prediction error decoding means the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering means to improve the quality of the output video before passing it for display and/or storing it as prediction reference for the forthcoming frames in the video sequence. [62] The motion information may be indicated with motion vectors associated with each motion compensated image block in video codecs. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder side) or decoded (in the decoder side) and the prediction source block in one of the previously coded or decoded pictures. H.264/AVC and HEVC, as many other video compression standards, a picture is divided into a mesh of rectangles, for each of which a similar block in one of the reference pictures is indicated for inter prediction. The location of the prediction block is coded as a motion vector that indicates the position of the prediction block relative to the block being coded. [63] In order to represent motion vectors efficiently those may be coded differentially with respect to block specific predicted motion vectors. The predicted motion vectors may be created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and/or co-located blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded/decoded picture can be predicted. The reference index may be predicted from adjacent blocks and/or or co-located blocks in temporal reference picture. Moreover, high efficiency video codecs may employ an additional motion information coding/decoding mechanism, often called merging/merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification/correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and/or co- located blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidate list filled with motion field information of available adjacent/co-located blocks. [64] H.264/AVC and HEVC, as many other video compression standards, a picture is divided into a mesh of rectangles, for each of which a similar block in one of the reference pictures is indicated for inter prediction. The location of the prediction block is coded as a motion vector that indicates the position of the prediction block relative to the block being coded. [65] A bitstream may be defined as a sequence of bits or a sequence of syntax structures. A bitstream format may constrain the order of syntax structures in the bitstream. [66] A syntax element may be defined as an element of data represented in the bitstream. A syntax structure may be defined as zero or more syntax elements present together in the bitstream in a specified order. [67] In some coding formats or standards, a bitstream may be in the form of a network abstraction layer (NAL) unit stream or a byte stream, which forms the representation of coded pictures and associated data forming one or more coded video sequences. [68] A NAL unit may be defined as a syntax structure containing an indication of the type of data to follow and bytes containing that data in the form of an RBSP interspersed as necessary with start code emulation prevention bytes. A raw byte sequence payload (RBSP) may be defined as a syntax structure containing an integer number of bytes that is encapsulated in a NAL unit. An RBSP is either empty or has the form of a string of data bits containing syntax elements followed by an RBSP stop bit and followed by zero or more subsequent bits equal to 0. [69] A NAL unit comprises a header and a payload. The NAL unit header indicates the type of the NAL unit among other things. [70] In some coding formats, such as AV1, a bitstream may comprise a sequence of open bitstream units (OBUs). An OBU comprises a header and a payload, wherein the header identifies a type of the OBU. Furthermore, the header may comprise a size of the payload in bytes. [71] The phrase along the bitstream (e.g., indicating along the bitstream) or along a coded unit of a bitstream (e.g., indicating along a coded tile) may be used in claims and described embodiments to refer to transmission, signaling, or storage in a manner that the "out-of-band" data is associated with but not included within the bitstream or the coded unit, respectively. The phrase decoding along the bitstream or along a coded unit of a bitstream or alike may refer to decoding the referred out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) that is associated with the bitstream or the coded unit, respectively. For example, the phrase along the bitstream may be used when the bitstream is contained in a container file, such as a file conforming to the ISO Base Media File Format, and certain file metadata is stored in the file in a manner that associates the metadata to the bitstream, such as boxes in the sample entry for a track containing the bitstream, a sample group for the track containing the bitstream, or a timed metadata track associated with the track containing the bitstream. [72] Video codecs may support motion compensated prediction from one source image (uni-prediction) and two sources (bi-prediction). In the case of uni-prediction a single motion vector is applied whereas in the case of bi- prediction two motion vectors are signaled and the motion compensated predictions from two sources are averaged to create the final sample prediction. In the case of weighted prediction, the relative weights of the two predictions can be adjusted, or a signaled offset can be added to the prediction signal. [73] In addition to applying motion compensation for inter picture prediction, similar approach can be applied to intra picture prediction. In this case the displacement vector indicates where from the same picture a block of samples can be copied to form a prediction of the block to be coded or decoded. This kind of intra block copying methods can improve the coding efficiency substantially in presence of repeating structures within the frame – such as text or other graphics. [74] The prediction residual after motion compensation or intra prediction may be first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and transform can in many cases help reduce this correlation and provide more efficient coding. [75] Video encoders may utilize Lagrangian cost functions to find optimal coding modes, e.g., the desired Macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor λ to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area: C = D + λR (Eq.1) Where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., Mean Squared Error) with the mode and motion vectors considered, and R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors). [76] Features and coding tools included in VVC include the following: ^ Intra prediction – 67 intra mode with wide angles mode extension – Block size and mode dependent 4 tap interpolation filter – Position dependent intra prediction combination (PDPC) – Cross component linear model intra prediction (CCLM) – Multi-reference line intra prediction – Intra sub-partitions – Weighted intra prediction with matrix multiplication ^ Inter-picture prediction – Block motion copy with spatial, temporal, history-based, and pairwise average merging candidates – Affine motion inter prediction – sub-block based temporal motion vector prediction – Adaptive motion vector resolution – 8x8 block-based motion compression for temporal motion prediction – High precision (1/16 pel) motion vector storage and motion compensation with 8-tap interpolation filter for luma component and 4-tap interpolation filter for chroma component – Triangular partitions – Combined intra and inter prediction – Merge with MVD (MMVD) – Symmetrical MVD coding – Bi-directional optical flow – Decoder side motion vector refinement – Bi-prediction with CU-level weight ^ Transform, quantization and coefficients coding – Multiple primary transform selection with DCT2, DST7 and DCT8 – Secondary transform for low frequency zone – Sub-block transform for inter predicted residual – Dependent quantization with max QP increased from 51 to 63 – Transform coefficient coding with sign data hiding – Transform skip residual coding ^ Entropy Coding – Arithmetic coding engine with adaptive double windows probability update ^ In loop filter – In-loop reshaping – Deblocking filter with strong longer filter – Sample adaptive offset – Adaptive Loop Filter ^ Screen content coding: – Current picture referencing with reference region restriction ^ 360-degree video coding – Horizontal wrap-around motion compensation ^ High-level syntax and parallel processing – Reference picture management with direct reference picture list signalling – Tile groups with rectangular shape tile groups [77] In H.266/VVC, the following block partitioning applies. Pictures may be divided into coding tree units (CTUs). A picture may also be divided into slices, tiles, bricks, and sub-pictures. CTU may be split into smaller CUs using quaternary tree structure. Each CU may be divided using quad-tree and nested multi-type tree including ternary and binary split. [78] There are specific rules to infer partitioning in in picture boundaries. [79] The redundant split patterns are disallowed in nested multi-type partitioning. [80] Some video coding tools perform filtering operations which convolve a set of reference samples with a set of filter parameters to output, for example, a predicted value for a certain sample in a picture. In some cases, the filter parameters may be predetermined or signaled in the bitstream. In other cases, such as when cross-component linear model (CCLM) or cross-component convolutional model (CCCM) prediction is used, the parameters are calculated using a set of reference samples in both encoder and decoder. Generally, calculation of such filter parameters involves inversion of an auto-correlation matrix, which is a computationally challenging operation. Also, when there are many filter parameters to be determined, the size of the auto-correlation matrix becomes large, which can cause numerical stability issues (overflows or underflows). [81] To reduce the cross-component redundancy, a cross-component linear model (CCLM) prediction mode may be used in the VVC, for which the chroma samples are predicted based on the reconstructed luma samples of the same CU by using a linear model as follows: pred_^(i, j) = α · rec_^′(i, j) + β (3-1) where pred_^ ⁽i, j⁾ represents the predicted chroma samples in a CU, and rec_^'⁽i, j⁾ represents the downsampled reconstructed luma samples of the same CU. [82] The CCLM parameters (α and β) are derived with at most four neighboring chroma samples and their corresponding down-sampled luma samples. Suppose the current chroma block dimensions are WxH, then W’ and H’ are set as − W’ = W, H’ = H when LM mode is applied; − W’ =W + H when LM-A mode is applied; − H’ = H + W when LM-L mode is applied. [83] The above neighboring positions are denoted as S[0, -1]…S[W’-1, - 1] and the left neighboring positions are denoted as S[-1, 0]…S[-1, H’-1]. Then the four samples are selected as − S[W’ / 4, −1 ], S[ 3 * W’ / 4, −1 ], S[ −1, H’ / 4 ], S[ −1, 3 * H’ / 4 ] when LM mode is applied and both above and left neighboring samples are available; − S[ W’ / 8, −1 ], S[ 3 * W’ / 8, −1 ], S[ 5 * W’ / 8, −1 ], S[ 7 * W’ / 8, −1 ] when LM-A mode is applied or only the above neighboring samples are available; − S[ −1, H’ / 8 ], S[ −1, 3 * H’ / 8 ], S[ −1, 5 * H’ / 8 ], S[ −1, 7 * H’ / 8 ] when LM-L mode is applied or only the left neighboring samples are available. [84] The four neighboring luma samples at the selected positions are down-sampled and compared four times to find two smaller values: x0A and x1A, and two larger values: x0B and x1B. Their corresponding chroma sample values are denoted as y0A, y1A, y0B and y1B. Then xA, xB, yA and yB are derived as [85] Finally, the linear model parameters are obtained according to the following equations: [86] Figure 1 shows an example of the location of the left and above samples and the sample of the current block involved in the CCLM mode. The division operation to calculate parameter α is implemented with a look-up table. To reduce the memory required for storing the table, the diff value (difference between maximum and minimum values) and the parameter α are expressed by an exponential notation. For example, diff is approximated with a 4-bit significant part and an exponent. Consequently, the table for 1/diff is reduced into 16 elements for 16 values of the significant as follows: DivTable [ ] = { 0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0 } [87] This would have a benefit of both reducing the complexity of the calculation as well as the memory size required for storing the needed tables. [88] Besides the above template and left template can be used to calculate the linear model coefficients (i.e., parameters) together, they can also be used alternatively in the other 2LM modes, called LM_A, and LM_L modes. [89] In LM_A mode, only the above template is used to calculate the linear model coefficients (i.e., parameters). To get more samples, the above template is extended to (W+H). In LM_L mode, only left template is used to calculate the linear model coefficients (i.e., parameters). To get more samples, the left template is extended to (H+W). [90] For a non-square block, the above template is extended to W+W, the left template is extended to H+H. [91] To match the chroma sample locations for 4:2:0 video sequences, two types of downsampling filter are applied to luma samples to achieve 2 to 1 downsampling ratio in both horizontal and vertical directions. The selection of downsampling filter is specified by a SPS level flag. The two downsampling filters are as follows, which are corresponding to “type-0” and “type-2” content, respectively. Rec_^′⁽^, ^⁾ [92] It is appreciated that only one luma line (general line buffer in intra prediction) is used to make the down-sampled luma samples when the upper reference line is at the CTU boundary. [93] This parameter computation is performed as part of the decoding process and is not just as an encoder search operation. As a result, no syntax is used to convey the α and β values to the decoder. [94] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. Those modes include five traditional intra modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). Chroma mode signalling and derivation process are shown in Table 1 in Figure 2a. Chroma mode coding directly depends on the intra prediction of the corresponding luma block. Since separate block partitioning structure for luma and chroma components is enabled in I slices, one chroma block may correspond to multiple luma blocks. Therefore, for Chroma DM mode, the intra prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited. [95] A single binarization table is used regardless of the values of sps_cclm_enabled_flag as shown in Table 2 in Figure 2b. In Table 2, the first bin indicates whether it is regular (0) or LM modes (1). If it is LM mode, then the next bin indicates whether it is LM_CHROMA (0) or not. If it is not LM_CHROMA, next 1 bin indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first bin of the binarization table for the corresponding intra_chroma_pred_mode can be discarded prior to the entropy coding. Or, in other words, the first bin is inferred to be 0 and hence not coded. This single binarization table is used for both sps_cclm_enabled_flag equal to 0 and 1 cases. The first two bines in Table 3- 4 are context coded with its own context model, and the rest bins are bypass coded. [96] In addition, in order to reduce luma-chroma latency in dual tree, when the 64x64 luma coding tree node is partitioned with Not Split (and ISP is not used for the 64x64 CU) or QT, the chroma CUs in 32x32/32x16 chroma coding tree node are allowed to use CCLM in the following way: - If the 32x32 chroma node is not split or partitioned QT split, all chroma CUs in the 32x32 node can use CCLM - If the 32x32 chroma node is partitioned with Horizontal BT, and the 32x17 child node does not split or use Vertical BT split, all chroma CUs in the 32x16 chroma node can use CCLM. In all the other luma and chroma coding tree split conditions, CCLM is not allowed for chroma CU. [97] The CCLM included in VVC is extended by adding three Multi-model LM (MMLM) modes. In each MMLM mode, the reconstructed neighboring samples are classified into two classes using a threshold which is the average of the luma reconstructed neighboring samples. The linear model of each class is derived using the Least-Mean-Square (LMS) method. For the CCLM mode, the LMS method is also used to derive the linear model. Figure 3a illustrates two luma-to-chroma models obtained for luma (Y) threshold of 17. Each luma- to-chroma model has its own linear model parameters α and β. As can be seen from Figure 3b, each luma-to-chroma model corresponds to a spatial segmentation of the content (i.e., they correspond to different objects or textures in the scene). [98] An improved version of cross-component prediction, known as convolutional cross-component model (CCCM), uses 2D filter kernel to derive the luma-to-chroma model. The filter coefficients are derived decoder-side using reconstructed set of input data and chroma samples. For the filter coefficient derivation, co-located reference sample areas (consisting of reconstructed luma and chroma samples) are defined for both luma and chroma as shown in Figure 4, yet any number of reference lines (that can be realized by both the encoder and decoder) can be used. Generally, reference samples can contain any chroma and luma samples that have been reconstructed by both the encoder and decoder. Once the reference samples have been determined, the filter coefficients can be derived, for example, using different types of linear regression tools such as ordinary least-squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operator. [99] The dimensions of the filter kernel can be for example 1x3 (1D vertical), 3x1 (1D horizontal), 3x3, 7x7 or any dimensions, and can be shaped (by selecting only a subset of all possible kernel locations) as a cross or a diamond or as any given shape. When referring to the samples within the filter kernel, the following notation is used: north (above), east (right), south (below), west (left) and center, as illustrated in Figure 5 using the letters N, E, S, W, C. Figure 5 illustrate a 3-tap vertical kernel 501, 3-tap horizontal kernel 502, 5-tap cross kernel 503 and 25-tap diamond kernel 504. [100] The overall method of reconstructing chroma samples using convolution between a decoder-side obtained filter kernel and a set of input data is referred to as convolutional cross-component model (CCCM) here. The following steps can be applied to perform a CCCM operation: - Define co-located reference areas over the luma and chroma components; - Down-sample the luma samples to match the chroma grid (optional); - Scan the luma and chroma samples of the reference area and collet available statistics (such as auto-correlation matrix and cross- correlation vector) based on the filter shape; - Solve the filter coefficients by minimizing squared-error (or any other metric) based on the available statistics (such as the auto-correlation matrix and cross-correlation vector); - Calculate a predicted chroma block by convolving the down-sampled luma samples with the filter kernel. [101] In the following, the (possibly down-sampled) luma samples are defined as a 2D array Y(x, y) indexed using horizontal x-coordinate and vertical y-coordinate. Also, the co-located chroma samples are defined as a 2D array C(x, y) and the filter kernel (i.e., coefficients) as 3x3 array F(i, j). On a sample level, the convolution between Y and F is defined as [102] When using other data terms, such as the non-linear square-root term, the appended convolution becomes where ^^{^} are filter coefficients that reside outside of the 2D filter kernel yet have been obtained as a part of the system of linear equations that were used to solve the 2D filter coefficients in Step 4 above. Similarly, the bias term can be added to the convolution with [103] Angular intra prediction (a.k.a. directional intra prediction) may be performed by extrapolating sample values from the reconstructed reference samples utilizing a given directionality. The reference samples may comprise the immediately neighboring sample row above and above-right of the current block (when available) and the immediately neighboring sample column on the left of the current block (when available), wherein availability may require decoding order earlier than that of the current block and presence in the same image segment, such as in the same tile. In order to simplify the process, all sample locations within one prediction block may be projected to a single reference row or column depending on the directionality of the selected prediction mode. A predicted sample within the block being encoded/decoded may be obtained by the following steps: - Projecting the location of a predicted sample to a location within a reference row or column by applying the selected prediction direction. The location within the reference row or column may have fractional sample accuracy, such as 1/32 pixel accuracy. - Interpolating a value for the sample location on the reference row or column from the reference samples at the reference row/column. [104] Multiple reference line (MRL) intra prediction uses more reference lines for intra prediction. In Figure 6, an example of four reference lines is depicted, where the samples of segments A and F are not fetched from reconstructed neighboring samples but padded with the closest samples from Segment B and E, respectively. HEVC intra-picture prediction uses the nearest reference line (i.e., reference line 0). In MRL, two additional lines (reference line 1 and reference line 3) are used. [105] The index of selected reference line (mrl_idx) is signaled and used to generate intra predictor. For reference line idx, which is greater than 0, only include additional reference line modes in MPM list and only signal mpm index without remaining mode. The reference line index is signaled before intra prediction modes, and Planar mode is excluded from intra prediction modes in case a nonzero reference line index is signaled. [106] MRL is disabled for the first line of blocks inside a CTU to prevent using extended reference samples outside the current CTU line. Also, PDPC is disabled when additional line is used. For MRL mode, the derivation of DC value in DC intra prediction mode for non-zero reference line indices are aligned with that of reference line index 0. MRL requires the storage of 3 neighboring luma reference lines with a CTU to generate predictions. The Cross-Component Linear Model (CCLM) tool also requires 3 neighboring luma reference lines for its own down-sampling filters. The definition of MLR to use the same 3 lines is aligned as CCLM to reduce the storage requirements for decoders. [107] The intra sub-partitions (ISP) divides luma intra-predicted blocks vertically or horizontally into 2 or 4 sub-partitions depending on the block size. For example, minimum block size for ISP is 4x8 (or 8x4). If block size is greater than 4x8 (or 8x4), then the corresponding block is divided by four sub- partitions. It has been noticed that the Mx12 (with M ≤ 64) and 128xN (with N ≤ 64) ISP blocks could generate a potential issue with the 64x64 VDPU. For example, and Mx128 CU in the single tree case has an M x 128 luma TB and ^ two corresponding ^ × 64 chroma TBs. If the CU uses ISP, then the luma TB will be divided into four Mx32 TBs (only the horizontal split is possible), each of them smaller than a 64 x 64 block. However, in the current design of ISP, chroma blocks are not divided. Therefore, both chroma components will have a size greater than a 32 x 32 block. [108] Analogously, a similar situation could be created with a 128 x N CU using ISP. Hence, these two cases are an issue for the 64 x 64 decoder pipeline. For this reason, the CU sizes that can use ISP is restricted to a maximum 64 x 64. All sub-partitions fulfil the condition of having at least 16 samples. [109] Matrix weighted intra prediction (MIP) method is an intra prediction technique in VVC. For predicting the samples of a rectangular block of width W and height H, matrix weighted intra prediction (MIP) takes one line of H reconstructed neighboring boundary samples left of the block and one line of W reconstructed neighboring boundary samples above the block as input. If the reconstructed samples are unavailable, they are generated as it is done in the conventional intra prediction. The generation of the prediction signal is based on the following three steps, which are averaging, matrix vector multiplication and linear interpolation as shown in Figure 7. [110] Merged mode with motion vector difference (MMVD) is to signal MVDs and a resolution index after signalling merge candidate. The merge list may comprise the following candidates: 1) Spatial MVP from spatial neighbor CUs 2) Temporal MVP from collocated CUs 3) History-based MVP from a FIFO table 4) Pairwise average MVP (using the candidates already in the list) 5) Zero MVs. [111] In Symmetric MVD, motion information of list-1 is derived from motion information from list-0 in bi-prediction case. [112] In Affine prediction, several motion vectors are indicated/signaled for different corners of a block, which are used to derive the motion vectors of sub- block. In affine merge, affine motion information of a block is generated based on the normal or affine motion information of the neighboring blocks. [113] In Sub-block-based temporal motion vector prediction, motion vectors of sub-blocks of the current block are predicted from a proper subblocks in the reference frame which are indicated by the motion vector of a spatial neighboring vector (if available). [114] In Adaptive motion vector resolution (AMVR), precision of MVD is signaled for each CU. [115] In BI-prediction with CU-level weight, an index indicated the weight values for weighted average of two prediction block. [116] Bi-directional optical flow (BDOF) refines the motion vectors in bi- prediction case. BDOF generates two prediction blocks using the signaled motion vectors. Then a motion refinement is calculated to minimize the error between two prediction blocks using their gradient values. The final prediction blocks are refined using the motion refinement and gradient values. [117] In HEVC, the bi-prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and/or using two different motion vectors. In VVC, the bi-prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals. This is called bi-prediction with CU-level weight (BCW): ^_bi-pred [118] Five weights are allowed in the weighted averaging bi-prediction, ^ ∈ {−2, 3, 4, 5, 10}. For each bi-predicted CU, the weight w is determined in one of two ways: 1) for a non-merge CU, the weight index is signaled after the motion vector difference; 2) for a merge CU, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width times CU height is greater than or equal to 256). For low-delay pictures, all 5 weights are used. For non-low-delay pictures, only 3 weights (w∈{3,4,5}) are used. [119] At the encoder, fast search algorithms are applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized as follows. For further details readers are referred to the VTM software and document JVET-L0646. When combined with AMVR, unequal weights are only conditionally checked for 1-pel and 4-pel motion vector precisions if the current picture is a low-delay picture. − When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode. − When the two reference pictures in bi-prediction are the same, unequal weights are only conditionally checked. − Unequal weights are not searched when certain conditions are met, depending on the POC distance between current picture and its reference pictures, the coding QP, and the temporal level. [120] The BCW weight index is coded using one context coded bin followed by bypass coded bins. The first context coded bin indicates if equal weight is used; and if unequal weight is used, additional bins are signalled using bypass coding to indicate which unequal weight is used. [121] Weighted prediction (WP) is a coding tool supported by the H.264/AVC and HEVC standards to efficiently code video content with fading. Support for WP was also added into the VVC standard. WP allows weighting parameters (weight and offset) to be signaled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weight(s) and offset(s) of the corresponding reference picture(s) are applied. WP and BCW are designed for different types of video content. In order to avoid interactions between WP and BCW, which will complicate VVC decoder design, if a CU uses WP, then the BCW weight index is not signaled, and w is inferred to be 4 (i.e., equal weight is applied). For a merge CU, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to both normal merge mode and inherited affine merge mode. For constructed affine merge mode, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index for a CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV. [122] In VVC, combined inter/intra prediction (CIIP) and bi-prediction with CU-level weight (BCW) cannot be jointly applied for a CU. When a CU is coded with CIIP mode, the BCW index of the current CU is set to 2, e.g., equal weight. [123] In VVC, when a CU is coded in merge mode, if the CU contains at least 64 luma samples (that is, CU width times CU height is equal to or larger than 64), and if both CU width and CU height are less than 128 luma samples, an additional flag is signaled to indicate if the combined inter/intra prediction (CIIP) mode is applied to the current CU. As its name indicates, the CIIP prediction combines an inter prediction signal with an intra prediction signal. The inter prediction signal ^_^^^^^ in the CIIP mode is derived using the same inter prediction process applied to regular merge mode; and the intra prediction signal ^_^^^^^ is derived following the regular intra prediction process with the planar mode. Then, the intra and inter prediction signals ^_^^^^^ , ^_^^^^^ are combined using weighted averaging, where the weight value is calculated depending on the coding modes of the top and left neighboring blocks as follows: – If the top neighbor is available and intra coded, then set isIntraTop to 1, otherwise set isIntraTop to 0; – If the left neighbor is available and intra coded, then set isIntraLeft to 1, otherwise set isIntraLeft to 0; – If (isIntraLeft + isIntraTop) is equal to 2, then wt is set to 3; – Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, then wt is set to 2; – Otherwise, set wt to 1. Figure 8 illustrates an example of top and left neighboring blocks used in CIIP weight derivation. [124] The CIIP prediction is formed as follows: [125] Local illumination compensation (LIC) is an inter prediction technique to model local illumination variation between current block and its prediction block as a function of that between current block template and reference block template. The parameters of the function can be denoted by a scale α and an offset β, which forms a linear equation, that is, α*p[x]+β to compensate illumination changes, where p[x] is a reference sample pointed to by MV at a location x on reference picture. Since α and β can be derived based on current block template and reference block template, no signaling overhead is required for them, except that an LIC flag is signaled for AMVP mode to indicate the use of LIC. [126] The local illumination compensation proposed in JVET-O0066 is used in ECM for uni-prediction inter CUs with the following modifications. ^ Intra neighbor samples can be used in LIC parameter derivation; ^ LIC is disabled for blocks with less than 32 luma samples; ^ For both non-subblock and affine modes, LIC parameter derivation is performed based on the template block samples corresponding to the current CU, instead of partial template block samples corresponding to first top-left 16x16 unit; ^ Samples of the reference block template are generated by using MC with the block MV without rounding it to integer-pel precision. [127] In bi-directional motion compensation the out-of-boundary (OOB) prediction samples are discarded and only the non-OOB predictors, when available, are used to generate the final predictor. The OOB prediction samples are such samples whose reference sample is at least partly located outside the reference picture. Specifically, let ^^^_^_^,^ and ^^^_^_^,^ denote the position of one prediction sample in one current block, ^^_^^^ ^^ ^_,^ and ^^_^_^,^ (x = 0,1) denote the MV of the current block; ^^^_^^^^^^^^, ^^^_^^^^^^^^^, ^^^_^^^^^^^ and ^^^_^^^^^^^^^^ are the positions of four boundaries of the picture. One prediction sample is regarded as OOB when at least one of the following conditions is satisfied: (^^^_^_^,^ + ^^_^^^ ^_,^) > (^^^_^^^^^^^^^+half_pixel), (^^^_^_^,^ + ^^_^^^ ^_,^) < (^^^_^^^^^^^^- half_pixel), (^^^_^ ^^ ^_,^ + ^^_^_^,^) < (^^^_^^^^^^^- half_pixel) where half_pixel is equal to 8 that represents the half-pel sample distance in the 1/16-pel sample precision. [128] After examining the OOB condition for each sample, the final prediction samples of one bi-directional block are generated as shown in the following algorithm A: If ^^^ is OO ^^ ^_,^ B and ^_^,^ is non-OOB ^ ^^^^^ ^^ ^,^ = ^_^,^ else if ^^^ is non-O ^^ ^_,^ OB and ^_^,^ is OOB ^ ^^^^^ = ^^{^^} ^,^ _^,^ else [129] In the final prediction, as shown, above, the first part (“if… else if…”) is a uni-predictive part of the bi-directional The bi- part of the bi-directional prediction is the second part (“else in which two uni-predictions are (weighted) averaged to form a true bi-prediction. [130] OOB checking process is also applicable when BCW is enabled. [131] Finally, it is to be noticed that this sample-adaptive bi-prediction process may only apply to prediction units for which at least a reference block is first detected as partially or entirely out-of-boundary. Thus, a block-level OOB criteria is first checked. If both prediction blocks are non-OOB, then the usual bi-prediction takes place. [132] In OOB-aware bi-directional inter-prediction only samples that are available in both prediction blocks (i.e.., motion compensated blocks P0 and P1) can be merged into a bi-prediction. For areas, where only one of the predictors (either P0 or P1) is available (the other one being out of the image boundaries), only samples belonging to the available uni-predictor can be used. Therefore, the final prediction of the block is a mixture of uni-prediction samples and bi-prediction samples. This can lead to artefacts and abrupt edges when uni-predicted samples have different brightness or texture characteristics compared to the bi-predicted samples. [133] The present embodiments are targeted to a linear filter for bi- prediction to adjust the uni-prediction part of the OOB-aware bi-prediction block to better match the bi-predictive part of said block. This means that the present embodiments aim to filter the uni-prediction parts based on how the bi-prediction looks. The filter coefficients are derived by minimizing the squared error of the bi-predictive part against a set of uni-prediction samples. [134] When OOB-handling is enabled, the inter-prediction block can become a mixture of bi-prediction and uni-prediction signals. This means that in OOB-aware prediction, the final bi-prediction block includes both a uni- prediction part and a bi-prediction part. Bi-prediction part is only available for an area where both blocks are within image boundaries. If an area in a block is out of image boundaries, then uni-prediction is used for that area. Figure 9 shows two uni-prediction blocks P0910 and P1920, wherein grey area of P1 920 indicates out-of-boundary area for P1920. The final bi-predictive block when OOB handling is enabled, is shown with reference 952 and part of the block 951 is predicted using only the P0 uni-prediction block. The present embodiments propose to filter the samples in said uni-prediction area based on how the bi-predictive part (i.e., P0+P1) looks with respect to the P0 (or P1) uni-prediction block. [135] The predicted samples values in the bi-predictive part P0+P1 900 may be calculated as (P0 + P1) / 2 or more generally as ^ ∙ ^0 + (1 − ^) ∙ ^1, where a is a weight. The part of P0 that overlaps with P0P1 can be defined as X, where P0P1 is defined as P0P1 = P0+P1. The part of P0P1 that overlaps with P0 is defined as Y, as shown in Figure 10. [136] X and Y can be vectorized as column vectors. Coefficients c which minimize ||^^ − ^||^{^} are determined. A is data matrix containing the column vector X and optionally other columns, such as bias term or spatial shifts of P0. Matrix A may also contain samples from a collocated block in another channel. For example, when a filter is applied on a chroma channel, one or more columns in matrix A may be populated with sample values from a luma block. The minimization problem can be solved for example with simple linear regression, LDL, decomposition, or Gaussian elimination. Once the coefficients c have been estimated, the non-overlapping part of P0 prediction is adjusted as ^^{^} _^ = ^_^^, where Af is the data matrix obtained over the non-overlapping area of P0 defined as Xf in Figure 10. [137] Alternative partitions can be used for filter derivation as shown in Figure 11, where only the boundary area (i.e., dashed area 1110) of the overlapping area is used for filter derivation, yet the filter is applied over the full Xf . [138] Additionally, several filters can be derived on sub-partitions and applied on their respective sub-partitions as shown in Figure 12. Figure 12 shows an example where two sets of filter coefficients are derived on overlapping areas 0, 1, and applied in their respective areas 0, 1 over Xf . [139] As the coefficients are derived using the P0P1/X/Y area, the resulting filter may be able to model the bi-prediction process best close to the border of Xf and X areas, and further away from the X block the accuracy of the filtering might reduce. Especially for larger blocks, the distance between some of the samples for which the filter is applied and the samples that were used to determine the filter coefficients becomes large, which may lead to reduced quality of the filtered samples. To alleviate this issue, the determined filter may be applied to only a selected subset of samples belonging to the Xf area. For example, the filter may be applied only if the distance to the border of Xf and X is less than N samples, where N can be a predetermined number, such as 4 or 8, or it can be determined in other means, for example, using the dimensions of the prediction block. [140] Alternatively, or in addition, to selecting a subset of the samples in the Xf area for filtering, the filter’s impact may be adjusted based on the distance from the border between Xf and X. This can be done, for example by blending the output of filter Afc with the input samples in the Xf area. For example, if the distance from the border between Xf and X is marked as d, the output of the filtering operation can be determined as follows: [141] As another example, some of the samples in the P0P1/X/Y area can also be filtered with the determined filter to smoothen the transition between bi-predicted area and uni-predicted area. Also in this case, the predicted samples close to the border between Xf and X can be determined as a weighted sum of two samples, for example a bi-predicted sample in P0P1 are and a sample obtained by filtering uni-predicted samples in the X area. [142] According to an embodiment, the use of the described OOB filtering can be inferred from the samples of the uni-predictive signals or from the merged bi-prediction signal. [143] According to an embodiment, the filter may have a simple model such as a*x+b or it may follow more complex 2D model where additional data terms are obtained as spatial shifts of the uni-prediction signal (resulting in 2D convolutional filter). [144] According to an embodiment, the filter may contain non-linear terms. [145] According to an embodiment, the filter coefficients can be derived for any set or configuration of samples that belong to the overlapping area of the OOB-aware bi-predictive signals. [146] According to an embodiment, more than one filter can be derived and applied. Additionally, each filter may use its own area for filter derivation, and each filter may be applied in its own area over the uni-prediction part of the OOB-aware block. [147] According to an embodiment, the filter output is blended with uni- predicted sample values. [148] According to an embodiment, the filter output is blended with uni- predicted sample values based on a distance from a border between a uni- predicated area and a bi-predicted area. [149] According to an embodiment, the area on which the filter coefficients are derived can be inferred from the samples of the uni-predictive signals or from the merged bi-prediction signal. [150] According to an embodiment, the area on which the filter coefficients are derived can be signaled at CU, PU, CTU, slice, frame or sequence level. [151] The method according to an embodiment is shown in Figure 13. The method generally comprises processing 1010 a video frame to predict a block according to prediction samples; determining 1020 when both prediction blocks for a block are within boundaries of an image, and applying a bi- prediction based on said prediction blocks for the block resulting in a bi- prediction block; determining 1030 when one of the prediction blocks is at least partially out of image’s boundaries, and applying a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; deriving 1040 filter coefficients by minimizing squared error between the bi- prediction block and the uni-prediction block; and filtering 1050 at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block. Each of the steps can be implemented by a respective module of a computer system. [152] An apparatus according to an embodiment comprises means for processing a video frame to predict a block according to prediction samples; means for determining when both prediction blocks for a block are within boundaries of an image, and means for applying a bi-prediction based on said prediction blocks for the block resulting in a bi-prediction block; means for determining when one of the prediction blocks is at least partially out of image’s boundaries, and means for applying a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; means for deriving filter coefficients by minimizing squared error between the bi- prediction block and the uni-prediction block; and means for filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block. The means comprises at least one processor, and a memory including a computer program code, wherein the processor may further comprise processor circuitry. The memory and the computer program code are configured to, with the at least one processor, cause the apparatus to perform the method of Figure 13 according to various embodiments. [153] An example of a data processing system for an apparatus is illustrated in Figure 14. Several functionalities can be carried out with a single physical device, e.g., all calculation procedures can be performed in a single processor if desired. The data processing system comprises a main processing unit 100, a memory 102, a storage device 104, an input device 106, an output device 108, and a graphics subsystem 110, which are all connected to each other via a data bus 112. [154] The main processing unit 100 is a conventional processing unit arranged to process data within the data processing system. The main processing unit 100 may comprise or be implemented as one or more processors or processor circuitry. The memory 102, the storage device 104, the input device 106, and the output device 108 may include conventional components as recognized by those skilled in the art. The memory 102 and storage device 104 store data in the data processing system 100. [155] Computer program code resides in the memory 102 for implementing, for example, a method as illustrated in a flowchart of Figure 14 according to various embodiments. The input device 106 inputs data into the system while the output device 108 receives data from the data processing system and forwards the data, for example to a display. The data bus 112 is a conventional data bus and while shown as a single line it may be any combination of the following: a processor bus, a PCI bus, a graphical bus, an ISA bus. Accordingly, a skilled person readily recognizes that the apparatus may be any data processing device, such as a computer device, a personal computer, a server computer, a mobile phone, a smart phone, or an Internet access device, for example Internet tablet computer. [156] Figure 15 illustrates an example of a video encoder, where In: Image to be encoded; P’n: Predicted representation of an image block; Dn: Prediction error signal; D’n: Reconstructed prediction error signal; I’n: Preliminary reconstructed image; R’n: Final reconstructed image ; T, T^-1: Transform and inverse transform; Q, Q^-1: Quantization and inverse quantization; E: Entropy encoding; RFM: Reference frame memory; Pinter: Inter prediction; Pintra: Intra prediction; MS: Mode selection; F: Filtering. Figure 16 illustrates a block diagram of a video decoder where P’n: Predicted representation of an image block; D’n: Reconstructed prediction error signal; I’n: Preliminary reconstructed image; R’n: Final reconstructed image; T^-1: Inverse transform; Q^-1: Inverse quantization; E^-1: Entropy decoding; RFM: Reference frame memory; P: Prediction (either inter or intra); F: Filtering. An apparatus according to an embodiment may comprise only an encoder or a decoder, or both. [157] The various embodiments can be implemented with the help of computer program code that resides in a memory and causes the relevant apparatuses to carry out the method. For example, a device may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the device to carry out the features of an embodiment. Yet further, a network device like a server may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the network device to carry out the features of various embodiment. [158] If desired, the different functions discussed herein may be performed in a different order and/or concurrently with other. Furthermore, if desired, one or more of the above-described functions and embodiments may be optional or may be combined. [159] Although various aspects of the embodiments are set out in the independent claims, other aspects comprise other combinations of features from the described embodiments and/or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims. [160] It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications, which may be made without departing from the scope of the present disclosure as, defined in the appended claims.

Claims

Claims: 1. An apparatus for encoding, comprising: means for processing a video frame to predict a block according to prediction samples; means for determining when both prediction blocks for a block are within boundaries of an image, and means for applying a bi- prediction based on said prediction blocks for the block resulting in a bi-prediction block; means for determining when one of the prediction blocks is at least partially out of image’s boundaries, and means for applying a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; means for deriving filter coefficients by minimizing squared error between the bi-prediction block and the uni-prediction block; means for filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni- prediction block.

2. The apparatus according to claim 1, further comprising means for indicating a use of filtering in samples of the uni-prediction block or bi- prediction block.

3. The apparatus according to claim 1 or 2, wherein the filter contains non- linear terms.

4. The apparatus according to claim 1 or 2 or 3, further comprising means for deriving the filter coefficients for a set of samples that belong to an overlapping area of the bi-prediction blocks.

5. The apparatus according to any of the claims 1 to 4, further comprising means for deriving more than one filter, wherein each filter uses its own area for filter derivation and is applied in its own area over the uni- prediction block.

6. The apparatus according to any of the claims 1 to 5, further comprising means for indicating an area on which the filter coefficients are derived in samples of the uni-predictive block or in the bi-prediction block.

7. The apparatus according any of the claims 1 to 6, further comprising means for signaling an area of which the filter coefficients are derived at a coding unit, a prediction unit, a coding tree unit, a slice, a frame or a sequence level.

8. A method comprising: processing a video frame to predict a block according to prediction samples; determining when both prediction blocks for a block are within boundaries of an image, and applying a bi-prediction based on said prediction blocks for the block resulting in a bi-prediction block; determining when one of the prediction blocks is at least partially out of image’s boundaries, and applying a uni-prediction based on the prediction block within image boundaries resulting in a uni- prediction block; deriving filter coefficients by minimizing squared error between the bi-prediction block and the uni-prediction block; filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block.

9. The method according to claim 8, further comprising indicating a use of filtering in samples of the uni-prediction block or bi-prediction block.

10.The method according to claim 8 or 9, wherein the filter contains non- linear terms.

11.The method according to claim 8 or 9 or 10, further comprising deriving filter coefficients for a set of samples that belong to an overlapping area of the bi-prediction block.

12.The method according to any of the claims 8 to 11, further comprising deriving more than one filter, wherein each filter uses its own area for filter derivation and is applied in its own area over the uni-prediction block.

13.The method according to any of the claims 8 to 12, further comprising indicating an area on which the filter coefficients are derived in samples of the uni-predictive block or in the bi-prediction block.

14.The method according any of the claims 8 to 13, further comprising signaling an area of which the filter coefficients are derived at a coding unit, a prediction unit, a coding tree unit, a slice, a frame or a sequence level.

15.An apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: process a video frame to predict a block according to prediction samples; determine when both prediction blocks for a block are within boundaries of an image, and apply a bi-prediction based on said prediction blocks for the block resulting in a bi-prediction block; determine when one of the prediction blocks is at least partially out of image’s boundaries, and apply a uni-prediction based on the prediction block within image boundaries resulting in a uni- prediction block; derive filter coefficients by minimizing squared error between the bi-prediction block and the uni-prediction block; filter at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block.