WO2026007136A1

WO2026007136A1 - Intra-prediction method for video coding involving template gradients and apparatus incorporating the same

Info

Publication number: WO2026007136A1
Application number: PCT/CN2024/104074
Authority: WO
Inventors: Alexey Filippov; Vasily Rufitskiy; Jacek Konieczny; Hongdong QIN; Tianyu DONG; Zhuowei XU
Original assignee: Shenzhen TCL New Technology Co Ltd
Current assignee: Shenzhen TCL New Technology Co Ltd
Priority date: 2024-07-05
Filing date: 2024-07-05
Publication date: 2026-01-08
Anticipated expiration: 2027-01-05
Also published as: WO2026007223A1

Abstract

A video coding intra-prediction method is provided. The method includes: acquiring a reference area of a current block; determining a size or a type of a gradient operator; estimating horizontal gradients and vertical gradients of samples within the reference area with the gradient operator; and selecting an intra prediction mode or updating a list of candidate prediction modes for the current block based on the horizontal gradients and the vertical gradients.

Description

INTRA-PREDICTION METHOD FOR VIDEO CODING INVOLVING TEMPLATE GRADIENTS AND APPARATUS INCORPORATING THE SAME

TECHNICAL FIELD

The present disclosure generally relates to encoding and decoding technology, and in particular to an intra-prediction method for video coding, an apparatus incorporating the same, and a computer readable medium.

BACKGROUND

It is possible to avoid signaling a selected intra prediction mode by deriving the intra prediction mode at both the encoder and decoder using previously encoded/decoded samples. Two techniques are known and are currently integrated into an Enhanced Compression Model (ECM) , i.e., Decoder-Side Intra Mode Derivation (DIMD) and Template-Based Intra Mode Derivation (TIMD) .

The DIMD technique is currently used as one of the intra prediction modes. In this technique, one or several intra prediction modes are determined based on estimation of Histogram of Gradients (HoG) in the template area located to the left, above-left, and above a current block.

A flowchart of HoG estimation is shown in FIG. 1. For each position “A” to “R” , samples of template area are convolved with 2 filter cores of one fixed edge operator. One of these filter cores is horizontal (it detects gradients for horizontally-aligned edges) and the other one is vertical (it detects gradients for vertically-aligned edges) . Results of these convolutions are correspondingly denoted as “horizontal gradient” Gx and “vertical gradient” Gy.

These gradients are used to update a bin of HoG by increasing a value inside the bin by the sum of the absolute values of the gradients. Index of the bin is determined based on a ratio of Gx to Gy (or vice versa, Gy to Gx) and a combination of gradients’ signs.

FIG. 2 illustrates steps of DIMD which are performed at the encoder and decoder sides. Block parameters input at stage 1001 includes the location of a current block. This information is used to check the availability of neighboring reconstructed samples at stage 1002. If at least either the template above or to the left of the current block is available, gradient parameters (such as the gradient direction i_Ang and the gradient magnitude i_AMP) are computed at stage 1002 to build a HoG. Since a 3x3 sobel edge operator is always used (i.e., the size of the edge operator is less than the minimal block size) , it suffices in the DIMD design adopted in the DIMD design adopted into the ECM reference software to ensure that a side of the current block is not located on a picture boundary. If this condition be met, at least four reconstructed samples are available along this side of the current block.

Accuracy of results generated through the conventional DIMD technique may depend on contents and properties (such as the amount of noise, smoothness, etc. ) of the current block and its template area.

SUMMARY

Accordingly, the present disclosure aims to provide a video coding intra-prediction method, an apparatus incorporating the same, and a computer readable medium.

A technical scheme adopted by the present disclosure is to provide an intra-prediction method for video coding. The method includes: acquiring a reference area of a current block; determining a size or a type of a gradient operator; estimating horizontal gradients and vertical gradients of samples within the reference area with the gradient operator; and selecting an intra prediction mode or updating a list of candidate prediction modes for the current block based on the horizontal gradients and the vertical gradients.

Another technical scheme adopted by the present disclosure is to provide an apparatus for encoding or decoding. The apparatus includes a processor and a memory. The memory is configured to store executable instructions that, when executed by the processor, cause the processor to perform the foregoing intra-prediction method.

Another technical scheme adopted by the present disclosure is to provide a computer readable medium storing executable instructions that, when executed by a processor, cause the processor to perform the foregoing intra-prediction method.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to clearly explain the technical solutions in the embodiments of the present disclosure, the drawings used in the description of the embodiments will be briefly described below. Obviously, the drawings in the following description are merely some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings may also be obtained based on these drawings without any creative work.

FIG. 1 shows a Histogram of Gradients (HoG) estimation method utilized in DIMD technique.

FIG. 2 shows steps of DIMD performed at either the encoder or the decoder for gradient calculation.

FIG. 3 illustrates neighboring samples used in DIMD for HoG estimation for blocks of different sizes.

FIG. 4 shows dependency of the size of the template of the size of a current block.

FIG. 5 shows multiple reference line prediction (MRLP) in ECM 13.0.

FIG. 6 shows an exemplary TMRL candidate list.

FIG 7 illustrates signaling of a TMRL candidate index.

FIG. 8 is a flowchart of an intra-prediction method according to an embodiment of the present disclosure.

FIG. 9 illustrates different positions for applying a gradient operator to samples within a reference area.

FIG. 10 shows comparison of template areas of TIMD and DIMD for small block sizes.

FIG. 11 illustrates spatial positions of the window of a 2x2 operator for the left side template of a current block.

FIG. 12 shows a position of gradient estimation for the case of 3x3 window of gradient operator.

FIG. 13 shows a position of gradient estimation for the case of 2x2 window.

FIG. 14 illustrates comparisons of template areas of TIMD and DIMD for large block sizes.

FIG. 15 illustrates a method for operator size selection based on block size according to an embodiment of the present disclosure.

FIG. 16 shows a case of different number of reference lines in the above and left template areas.

FIG. 17 shows a case for analyzing the sizes of the blocks that underlie the template area.

FIG. 18 illustrates subblocks of the current block which are utilized to determine smoothness of the current block.

FIG. 19 illustrates regions in the reference area where blocking artifacts may occur.

FIG. 20 illustrates positions within the reference area for applying a 5x5 edge operator.

FIG. 21 shows a template processing scenario with edge operators of 2x2, 3x3, and 5x5 sizes.

FIG. 22 shows a template processing scenario with edge operators of 2x2, 3x3, 4x4, and 5x5 sizes.

FIG. 23 shows a template processing scenario with edge operators of 3x3 and 5x5 sizes.

FIG. 24 shows a template processing scenario with edge operators of 2x2 and 4x4 sizes.

FIG. 25 shows a case where non-adjacent DIMD with a 3x3 edge operator is applied to reference line 5.

FIG. 26A shows a case where non-adjacent DIMD with a 5x5 edge operator is applied to reference line 5.

FIG. 26B shows a case where non-adjacent DIMD with a 3x3 edge operator is applied to reference line 5 and its adjacent area.

FIG. 26C shows a case where non-adjacent DIMD with a 5x5 edge operator is applied to reference line 5 and its adjacent area.

FIG. 27 is a flowchart of intra prediction techniques (MPM and TMRL) , which can utilize non-adjacent reference lines.

FIG. 28 is a flowchart of a non-adjacent DIMD method used as a standalone intra-prediction technique.

FIG. 29 shows a scenario where a TMRL candidate list is updated with non-adjacent DIMD modes.

FIG. 30 illustrates a machine learning method for determining parameters of edge operators.

FIG. 31 shows a training procedure when coding a video sequence in Random Access configuration.

FIG. 32 shows a scenario where the template area of the current block is pre-filtered with no padding.

FIG. 33 shows a scenario where the template area of the current block is pre-filtered with padding.

FIG 34 is a flowchart of steps of obtaining gradient values of Gx and Gy for further building a histogram of gradients according to an embodiment of the present disclosure.

FIG. 35 shows a schematic diagram of an apparatus for encoding or decoding according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

The disclosure will now be described in detail with reference to the accompanying drawings and examples. Apparently, the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

The present disclosure relates to an intra-prediction method for video coding involving template gradient. For better understand of embodiments of the present disclosure, related art is introduced below.

Decoder-side Intra Mode Derivation (DIMD)

A decoder-side intra mode derivation (DIMD) approach was proposed in JVET-C0061. DIMD performs a gradient analysis on samples (e.g., luma samples) of a template that comprises neighbouring samples of current block. The template is L-shaped and has a width of 3 samples. When DIMD is in effect for a current block, the encoder and decoder calculate horizontal and vertical gradients inside the template to determine prevailing angular directions. In ECM, the encoder and decoder apply a 3x3 Sobel edge operator to positions of a 3x3 window within a template. The 3x3 Sobel gradient filter are defined as follows:

These gradients are further used to update histogram of gradients (HoG) . The maximum values of HoG indicate the prevailing angular directions inside a template. An encoder may indicate a flag that for the current block one or several prevailing directions could be used to determine one or several intra prediction modes that are used to intra-predict the current block.

A flowchart of obtaining HoG for the DIMD tool is shown in FIG. 1 and FIG. 2. It could be observed that in the current ECM-12.0 design, size of template does not depend on the block size (see FIG. 3) .

An intra direction corresponding to each pair of determined horizontal and vertical gradients Gx and Gy is determined as follows:

An intra prediction mode is obtained from the angle Θ (e.g., one of the 67 intra prediction modes defined in the H. 266 VVC standard) . A magnitude “G” for the pair of gradients and the corresponding intra prediction mode is determined according as follows:
|G|=|G_x| + |G_y|

When constructing a HoG, a bin of the histogram to be updated is obtained by taking the value of the determined intra prediction mode, and the height of the bin (i.e., a value associated with this bin) is updated using the determined magnitude |G|.

Template-based Intra Mode Derivation (TIMD)

A template-based intra mode derivation (TIMD) method approach was proposed in JVET-V0098. In this method template cost is calculated for modes of the most probable modes (MPM) list. The template cost is obtained for the neighboring areas of previously reconstructed samples. One of these areas are above the current block and the other one is to the left of the current block.

For each mode of the MPM list, a sum of the template costs for the template areas is calculated, and the mode that provides the smallest sum of template costs is selected as a TIMD mode. The encoder may indicate a flag that for a block TIMD mode is selected, instead of signaling this intra prediction mode explicitly. On the decoder side, based on this indication, the intra prediction mode could be restored by the same check of the MPM list that was performed on the encoder side. Size of a TIMD template is determined in accordance with the size of the block (see FIG. 4) .

Position dependent intra prediction combination (PDPC) is included in the derivation of the TIMD mode.

For each generated prediction, the encoder and decoder may then determine a template cost that is estimated as a prediction error based on a difference between the prediction samples determined for the intra prediction mode and the reconstructed samples of template. For example, a sum of squared differences (SSD) , a sum of absolute differences (SAD) , or a sum of absolute transformed differences (SATD) could be used as a template cost metric. The selected intra prediction mode may be referred to as the “TIMD intra prediction mode” and may be selected from the larger set of intra prediction modes (e.g., the set of 129 angular intra prediction modes) available to TIMD.

It should be noted that other selection criteria may be used to select an intra prediction mode from the applied intra prediction modes. In another example, the encoder and decoder may select the two intra prediction modes, from the applied intra prediction modes, that result in the first and second smallest predictions errors for a template area in a process referred to as “TIMD fusion. ” The encoder and decoder may use the two intra prediction modes by predicting current block using each of the two intra prediction modes and computing a final predictor based on a weighted average of each prediction determined using the two intra prediction modes.

Sizes of the template areas used in TIMD and DIMD methods are different from each other. Moreover, in ECM design, these two methods are alternative to each other: either TIMD or DIMD method is used to get one or several intra prediction modes for a current block, but not the both methods.

Template-based Multi-Reference Line Intra prediction (TMRL)

FIG. 5 depicts the design of multiple reference line prediction adopted into the ECM reference software. Reference line 0 is adjacent for a block to be predicted whereas reference lines 1, 3, 5, 7, and 13 are non-adjacent. Description of Template-based multiple reference line intra prediction (TMRL) mode could be found in the Algorithm description of Enhanced Compression Model 12 (ECM 12) , document JVET-AG2025.

Template-based multiple reference line intra prediction (TMRL) mode combines reference line and prediction mode together and uses a template matching method to construct a list of candidate combinations. An index to the candidate combination list is coded to indicate which reference line and prediction mode is used in coding the current block. The TMRL mode extends reference line candidate list and the intra-prediction-mode candidate list. The extended reference line candidate list is {1, 3, 5, 7, 12}.

The TMRL candidate is constructed as follows. There are 5x10=50 combinations of the extended reference line and the allowed intra-prediction modes for a block. Since the extended reference line starts from reference line 1, the area covered by reference line 0 is used for template matching. The SAD costs over the template area are calculated between the predictions (generated by 50 combinations) and the reconstructions. The 20 combinations with the least SAD cost are selected in an ascending order to form the TMRL candidate list.

A TMRL candidate list is shown in FIG. 6. Each element of the TMRL candidate list comprises the template cost, reference line and intra prediction mode. Out of the 50 candidates, only 20 combinations that provide minimum template cost are selected. These 20 candidates are sorted in ascending order by the cost values.

Indication of the selected index is performed by a Rice code (see FIG. 7) , that is composed of:1) a prefix code, which is a truncated unary code; and 2) a fixed-length suffix code. In ECM-13, suffix length (number of bins) does not depend on the prefix value and is set equal to 2 bins. Prefix may encode values within a range [0, 4] .

Reference sample availability marking process

In H. 266/VVC (prior art) reference sample availability marking process is invoked to determine which of the neighboring samples are reconstructed before the prediction of the current block, and thus could be used for intra prediction. The neighboring samples that are not reconstructed before the prediction of the current block, or the neighboring samples that are outside of the current slice are marked up as not available ones.

Contents of the H. 266/VVC related to the reference sample availability marking process, as well as all the relevant and dependent parts, are hereby incorporated by reference into this disclosure.

Existing DIMD design uses a predefined 3x3 Sobel edge operator that excludes any adaptivity to the content of neighboring reconstructed samples. This approach can have various issues and restrictions in terms of accuracy of the gradients determined by applying the edge operators.

The most fundamental assumption idea behind the usage of DIMD templates is that they are representative for a current block. This assumption is based on the spatial correlation between neighboring reconstructed samples and samples of the current block. Thus, the accuracy of deriving intra prediction modes strongly depends on whether the requirement for the representativeness of the templates is met or not. In the state-of-the-art DIMD design, the number of templates samples can exceed the number of samples within the current blocks of smaller sizes such as 4x4, 4x8, and 8x4 so that the samples in the template and in the current block might be not that strongly correlated with each other.

Sobel edge operator, that is currently fixed, could provide less accurate results than some other edge operator. Depending on the content and its properties, such as the amount of noise, smoothness, etc. More accurate HoG estimation could be done with an edge operator that is different from Sobel, e.g.: Prewitt, Scharr, Kirsch, etc. In particular, noise can harm the accuracy of gradient estimation which results in incorrectly derived intra-prediction modes and their parameters (e.g., fusion weights) and, hence, a lower compression performance of the DIMD technique.

Larger blocks tend to localize smoother content where the magnitudes of gradients are not as high as for blocks which localize edges. Moreover, blocks with smooth content might be more sensitive to noise. Thus, the accuracy of gradient estimation can suffer from sample deviation within smaller templates because their size might be insufficient to collect representative statistical data on gradients with small magnitudes.

In the present disclosure, it is proposed to use different edge detection operators when estimating Histogram of Gradients (HoG) , and the size of an edge operators’ filter core and type of the filter core (e.g., Sobel, Prewitt, Scharr, Kirsch, etc. ) is adaptively selected.

FIG. 8 illustrates a flowchart of an intra-prediction method for video coding according to an embodiment of the present disclosure. The method may be applied in an encoding or decoding process. As shown in FIG. 8, the method includes operations described in blocks S201 to S205, that is: acquiring a reference area of a current block; determining a size or a type of a gradient operator; estimating horizontal gradients and vertical gradients of samples within the reference area with the gradient operator; and selecting an intra prediction mode or updating a list of candidate prediction modes for the current block based on the horizontal gradients and the vertical gradients.

In S201, a reference area of a current block is acquired.

To process a current block, a reference area may be firstly determined. The method for determining the reference area may be in accordance with the selected prediction method such as DIMD, TMRL etc. For example, the reference area may include one or more reference lines adjacent or non-adjacent to the current block.

In some embodiments, availability check of neighboring reconstructed samples of the current block may be performed. The check of availability may be performed in accordance with, for example, H. 266/VCC specification “8.4.5.2.8 Reference sample availability marking process” . Since the size of edge operators to be applied to the template areas can exceed the size of the minimal block (i.e., 4) , we determine how many neighboring reconstructed samples are available for both the top and left sides of the current block. In some prediction methods, the maximum depth, i.e., the number of reference lines in the neighboring area of the block being predicted to be checked for their availability should be greater or equal to the maximum size of the edge operators which can be applied to these reconstructed samples. The reference area may include these areas with available samples.

In S202, a size or a type of a gradient operator is determined.

In this stage, the properties/features such as the size and type of a gradient operator may be selected. Multiple gradient operators with different sizes and types may be applied. The gradient operator may be a 3 by 3, 2 by 2 or 5 by 5 operator. The filter cores of the gradient operator may include, for example by not limited to, Sobel, Prewitt, Scharr, Kirsch, etc. In one embodiment, the size of the current block may be utilized to determine the size and/or the type of the gradient operator. In another embodiment, properties of the current block and/or the reference area of the current block such as smoothness may be utilized to determine the size and/or the type of the gradient operator.

In S203, horizontal gradients and vertical gradients of samples within the reference area are estimated with the gradient operator.

After the gradient operator is determined, it can be utilized for acquiring horizontal gradients and vertical gradients of samples within the reference area. The method for calculating horizontal and vertical gradients has been introduced in the section “Decoder-side Intra Mode Derivation” and will not be repeated herein.

FIG. 9 shows two exemplary reference areas. In one embodiment, gradients at all central samples within the reference area are estimated. For example, gradients at positions from A to M as shown in the diagram (A) may be determined. In another embodiment, gradients at some central samples within the reference area are estimated. For example, gradients at positions from B to L as shown in the diagram (A) may be determined while gradients at positions A and M are not calculated. In some embodiments, when the reference area includes one reference line as shown in the diagram (B) of FIG. 9, gradients at all samples from A to M may be determined by considering their adjacent samples (i.e., samples in the squares with the dotted outlines) .

In S204, an intra prediction mode is selected based on the horizontal gradients and the vertical gradients.

As explained in the section “Decoder-side Intra Mode Derivation” , the horizontal gradients and the vertical gradients acquired in S203 may be utilized to construct/update the histogram of gradients (HoG) , and therefore, to determine which intra prediction mode should be selected from, for example, the 67 intra prediction modes defined in the H. 266 VVC standard.

In S205, a list of candidate prediction modes is updated for the current block based on the horizontal gradients and the vertical gradients.

The HoG determined based on the horizontal gradients and the vertical gradients may also be utilized to select one or more intra prediction modes from. If the video coding process employs a method for which a list of prediction candidates is provided (e.g., TIMD, TMRL) , these intra prediction modes selected based on the HoG may be utilized to update the existing list. For example, these intra prediction modes selected based on the HoG may be added to the existing list. Alternatively, these intra prediction modes may be utilized to replace some prediction candidates within the existing list.

According to the present disclosure, the size or the type of the gradient operator may be adaptively determined based on, for example, properties of the current block and/or the reference area. Therefore, implementation of the present disclosure may enhance the flexibility of the DIMD method, potentially leading to more accurate prediction results for blocks with varying contents or properties.

The present disclosure provide gradient operators with different sizes. In some embodiments, the size of the gradient operator may be determined based on a size of the current block to be process. Introduction for methods for determining the size of the gradient operator is given in the following sections.

DIMD operator size selection based on block size (Small block)

As shown in FIG. 10, for blocks that comprise small number of samples (e.g., 4x4, 4x8 and 8x4) , a smaller window size (e.g., 2x2) could be used. Specifically, when the size of the current block is less than a first threshold, the size of the gradient operator may be determined as 2 by 2. The first threshold may be related to an amount of samples within the current block, or alternatively to a width or height of the current block. The first threshold may be pre-defined at both the encoder and decoder sides.

The size of the edge operator (gradient operator) window used in the gradient determination (DIMD operator) may be defined based on the size of the block. Moreover, the reference area (e.g., reference lines used in the DIMD template) may also be determined based on the size of the block and/or the size of the edge operator. Examples of such dependency are shown in Table 1 and Table 2.

Table 1. Dependency of the number of reference lines used in the DIMD template and the size of the edge operator window on the block size

Table 2. Dependency of the number of reference lines used in the DIMD template and the size of the edge operator window on the block size (alternative example)

The difference of the example shown in Table 2 compared to that of Table 1 is that for blocks with an area of 32 samples (i.e., 4x8 and 8x4) , the number of reference lines used in the DIMD template to estimate the gradient value is equal to 3, while the size of the edge operator window is 2x2. Spatial positions for applying the window for the left side template of a current block are shown in FIG. 11. Spatial positions for applying the window for the above side template of a current block could be selected similarly to the positions of the left template.

The gradient operator includes a horizontal operator F_x and a vertical operator F_y. An example of a 2x2 isotropic gradient operator is defined as:

Another example of a 2x2 Roberts Cross gradient operator is defined as:

A distinctive feature of this embodiment is that the number of columns and rows in the matrices that define filters’ cores is even (i.e., non-odd) . When applying such an operator to some 2x2 window of template samples, the gradient estimation is obtained for a fractional (half-pel) position inside this window. However, HoG estimation does not require exact location of the estimated gradient, so even dimensions of filter cores could be utilized.

FIG. 12 illustrates how gradient parameters for a 3x3 window of an edge operator are estimated. It could be noticed that gradient is estimated in a spatial position that coincides with the center of a 3x3 window.

When the width and the height of a window of a gradient operator comprises an even (non-odd) number of samples, a gradient estimation position does not coincide with a spatial position of the central element of a gradient operator window. For the case of even width and even height of a gradient operator window, a spatial position where the gradient is estimated falls in between integer positions of samples, i.e. the half-pel position between the samples that are closest to the center of the gradient operator window.

A 2x2 window of a gradient operator is an example of an even-sized width and height window. A position of the point where the gradient is estimated is in the middle between the 2 rows and the 2 columns, as shown in FIG. 13.

This non-integer position of the gradient estimation point may be a problem for edge detection tasks, because exact integer-pel position of an edge is undefined, since a gradient point position has no one-to-one correspondence with a position of a picture sample. However, when gradient estimation is a part of a histogram estimation process, there is no need to determine the exact location of and edge inside the picture. Hence, for the purpose of determination of the prevailing directions, even-sized widths and heights of an edge operator window are suitable.

It could be noticed, that the 2x2 size of a window of a gradient operator makes it possible to define a 2-lines width template area of DIMD, which is consistent with TIMD.

The implementation of the present embodiment may reduce the size of the edge operator window, and may reflect a better spatial correlation of DIMD template with small blocks.

DIMD operator size selection based on block size (Large block)

As shown in FIG. 14, for blocks that comprise large number of samples, a larger window size could be used. Specifically, when the size of the current block is larger than a second threshold, the size of the gradient operator may be determined as 5 by 5. It may be noticed that in this example, DIMD reference area is also aligned with TIMD.

The second threshold may be related to an amount of samples within the current block, or alternatively to a width or height of the current block. The second threshold may be pre-defined at both the encoder and decoder sides.

Large blocks typically have smoother content compared to smaller blocks. In this embodiment for large blocks more samples are used to obtain more reliable values of gradient magnitudes. For example, horizontal and vertical filters of a 5x5 Sobel edge operator could be defined as follows:

Another example of a 5x5 Prewitt edge operator may be defined as:

Another example of a 5x5 Scharr edge operator may be defined as:

The size of the edge operator (gradient operator) window used in the gradient determination (DIMD operator) may be defined based on the size of the block. Moreover, the reference area (e.g., reference lines used in the DIMD template) may also be determined based on the size of the block and/or the size of the edge operator. An example of such dependency is shown in Table 3 and Table 4.

Table 3. Dependency of the number of reference lines used in the DIMD template and the size of the edge operator window on the block size

In another example, the number of reference lines used in the DIMD template and the size of the edge operator window depend on the minimum side length (width or height) of a current block. This dependency is shown in Table. 4.

Table 4. Dependency of the number of reference lines used in the DIMD template and the size of the edge operator window on the minimum block side

In another example, the number of reference lines used in the DIMD template and the size of the edge operator window depend on the maximum side length (width W or height H) of a current block and the block aspect ratio defined as RASP= abs (log2W -log2H) . This dependency is exemplarily shown in Table 5.

Table 5. Dependency of the number of reference lines used in the DIMD template and the size of the edge operator window on the maximum block side

Implementation of the present embodiment may lead to a better accuracy of angular direction estimation. This effect is achieved by increasing the size of a filter core and thus increasing the number of reconstructed neighbor samples that are used to estimate gradient value.

FIG. 15 illustrates a flowchart for DIMD operator size selection based on block size. In some embodiments, both methods for small block and large blocks as described in the foregoing two sections may be adopted. In other embodiments, either the method for small block or the method for large block is adopted. Tables 6 to 8 illustrate several examples of a combination of criteria to determine the size of the DIMD edge operator.

Table 6. Dependency of the number of reference lines used in the DIMD template and the size of the edge operator window on the block size in one example

Table 7. Dependency of the number of reference lines used in the DIMD template and the size of the edge operator window on the block size in another example

Table 8. Dependency of the number of reference lines used in the DIMD template and the size of the edge operator window on the block size in yet another example

It’s understood that other combinations of criteria to determine the number of reference lines used in the DIMD template and the size of the edge operator window are also possible, including especially criteria based on maximum block size, number of samples in the block or their combinations.

Reference lines in the left and above template areas

In some embodiments, the gradient operator includes a first operator for a left part of the reference area and a second operator for a top (above) part of the reference area. The size of the first operator may correspond to the number of reference lines in the left part of the reference area while the size of the second operator may correspond to the number of reference lines in the top part of the reference area.

In some embodiments, the number of reference lines in the left and above template areas may be different, as shown in FIG. 16. Correspondingly, the size of the first operator may be different from the size of the second operator. One of the examples when the number of reference lines is different is the case when the number of available reference lines is limited by a position of the current block with regard to the slice boundaries. Note that templates might stick out of the block sides as shown in FIG. 16. The number of such template samples might depend on the size of the edge operator used to build HoG. For example, FIG. 16 illustrates a case where the number of template samples, which are not aligned with the block size, is different for top and left templates. For the top template, just 1 sample column sticks out of a current block as the edge operator of 3x3 size is applied to the top template, whereas the height of the left template exceeds the height of the current block by 2 sample rows because the edge operator of 5x5 size is applied to the top template. In general, the number of samples that sticks out N_SO can be computes as:
N_SO=L_EO＞＞1

where L_EO is the size of the edge operator used to calculate gradients, “>>” is a right shift operation.

In this embodiment, gradients in the left and in the above templates may be estimated using operators of different sizes, and these different operators may be normalized.

In this case, the norm of a filter S_N×M for an N×M matrix is estimated using the following equation:

Filter cores of the gradient operators could be normalized in advance using a multiplier Q_N×M:

F_norm (i, j) =Q_N×M·F (i, j)

The value of 2·P is a precision constant, that could be predefined to some specific value, e.g. 1024.

In a particular example, when a template area has 3 reference lines, a 3x3 gradient operator size is used. When a template area has 5 reference lines, a 5x5 gradient operator size is used.

The steps of this embodiment are as follows:

● In step 1, the number of reference lines is determined in the left and above template areas. This could be performed based on the availability of reconstructed samples above and to the left of the current block.

● In step 2, the filter cores of the gradient operators for the above and left template areas are determined.

● In step 3, the filter cores are normalized as disclosed above.

● In step 4, HoG is constructed using the steps of the prior-art.

Usage of window sizes and different filter cores of edge operators

In some embodiments, different filter cores may be selected for HoG estimation of a block.

For example, a 3x3 Prewitt operator may be selected, which is defined as:

In another example, a 3x3 Scharr operator may be selected, which is defined as:

In some embodiments, selection of the gradient operator may be determined based on statistical properties such as smoothness of sample distribution within the reference area. For a smoother content, a stronger edge operator may be selected. Smoothness of the content inside a template area may be implicitly estimated by analyzing the sizes of the blocks that underlie the template area (see FIG. 17) .

The steps of these embodiments are as follows.

● In step 1, availability of templates is determined. When reconstructed samples are not available for the both left and above sides of the current block, no further steps are performed.

● In step 2, the sides of the current block are subdivided into a group of subblocks (e.g., 4x4 blocks) . For each subblock, an area of a neighboring block is determined (see FIG. 18) .

● In step 3, each subblock is classified to being smooth or non-smooth by comparing a property of a neighboring block that comprises the subblock with a predefined threshold. For example, when a shorter side (width or height) of the neighboring block is larger than 8, the subblock is classified as a smooth one. Otherwise, a subblock is classified as a non-smooth one.

● In an alternative example, when the area of the neighboring block is larger than 64, the subblock is classified as a smooth one. Otherwise, a subblock is classified as a non-smooth one.

● In step 4, a number of the smooth subblocks is calculated. If the number of smooth subblocks is greater than a predetermined portion of a total number of subblocks, the current block is considered as smooth one. Otherwise, it is considered as a non-smooth one. For example, when more than half of the total number of subblocks are smooth, the block is considered as a smooth one.

In an exemplary embodiment, for smooth blocks Scharr operator is used to estimate gradients in HoG construction of the DIMD process. For non-smooth blocks, Prewitt operator is used in this operation. Table 9 illustrates exemplary gradient operators of different types and sizes that may possibly be adopted in different embodiments of the present disclosure.

Table 9. Edge operators of different types and sizes

Usage of activity function to determine an edge operators’ window size and a filter core

In some embodiments, the size and/or the filter core of the gradient operator may be determined based on: 1) normalized horizontal and vertical gradients of samples within the reference area which are acquired with a predefined operator; or 2) a statistical moment of samples within the reference area. Detailed explanation will be given below.

Activity function is used to determine an edge operators’ window size and a filter core. This activity function could be specified as a mean value of gradients calculated in the template area. By comparing the activity function value with some predetermined threshold, a type of content could be estimated, i.e. whether it is smooth or not. Therefore, a more proper edge operator could be selected. The steps of this embodiment are as follows:

In step 1, availability of templates is determined. When reconstructed samples are not available for the both left and above sides of the current block, no further steps are performed.

In step 2, horizontal and vertical gradients are calculated for a set of gradient operator window positions within the template area. In this step, a 2x2 gradient operator specified in the section “DIMD operator size selection based on block size (small block) ” may be utilized.

In step 3, a sum of absolute values of horizontal gradients S_y and a sum of absolute values of vertical gradients S_x is calculated.

In step 4, the sums obtained in step 3 are normalized to the total number P of different positions of the gradient operator window within the template area that were used in step 2:

In step 5, an activity function value is determined based on the normalized sums of gradientsandFor example, an average value of the two normalized sums may be used:

In another example, the maximum of the two sums can be used to determine the value of the activity function

In another example, the minimum of the two sums can be used to determine the value of the activity function

In another example, the activity function could be determined from the statistical properties of samples of the template area. In an exemplary embodiment, a standard deviation is estimated in the template area R

In another exemplary embodiment, a standard deviation is separately estimated for the top and left template areas (R_T and R_L, respectively) :

Resulting activity function value is obtained as a maximum of the two activity functions: the one is calculated for the top reference area and another one is calculated for the left reference area:

In another example, resulting activity function value is obtained as an average of the two activity functions: the one is calculated for the top reference area and another one is calculated for the left reference area:

In another example, resulting activity function value is obtained as a linear combination of the two activity functions based on the number of samples in the top ad above reference areas:

In another example, the value of activity function could use a difference between maximum and minimum sample value within the template area

The value of the multiplier k≤1 could be equal to, e.g., It is understood, in this example activity function could be estimated separately for the top and left template areas and the resulting activity function value could be calculated from these two estimations using the formulas above.

In addition, the above-described embodiments could use more than 2 template areas, including but not limited to: top-left template area, top-right template area, bottom-left template area.

In step 6, the gradient operator is determined based on the value of the activity function. For example, the determination could be performed by comparing the activity function with a predetermined threshold. Note that the value of the predetermined threshold could be defined with regard to the quantization parameter (QP) specified for the current block, or for the slice to which the current block belongs to. One of the possible ways to determine the gradient operator based on the value of the activity function is shown in Table 10.

Table 10. Dependency of the gradient operator used in DIMD HoG estimation on the activity function value

Note that the present embodiment may be applied independently to the left and to the above template areas.

Handling blocking artifacts in the template area

In some embodiments, the method shown in FIG. 8 may further include: determining positions of boundaries between different coding nits; and excluding points that intersect the boundaries from the samples within the reference area.

Although the use of smaller-size edge operator might improve the accuracy of HoG due to better spatial correlation, especially, for smaller blocks such as 4x4, 4x8, and 8x4, the HoG gets more sensitive to blocking artifacts appearing near the top-left corner of a current block as shown in FIG. 19. To alleviate this problem, we can omit the samples corresponding to these regions with blocking artefacts while handling the top and left templates of the blocks with an edge operator (for example, of 2x2 size) . The following steps may be performed:

● Determining availability of the top-left corner of the template area;

● Determining positions of boundaries between the coding units; and

● Collecting the values of local gradients at such window positions in the template area that do not intersect block boundaries that underlie the template area.

In an alternative embodiment, a boundary strength is determined for the boundary areas (see “Region of blocking artifacts” in FIG. 19) . This could be accomplished by applying the process “8.8.3.5 Derivation process of boundary filtering strength” described in the H. 266/VVC video coding.

Besides, in accordance with “8.8.3.6.6 Decision process for a luma sample” for samples on the block boundaries, decision value may be determined.

When the decision for all the samples of the “region of blocking artifact” is equal to 1, boundary position is not skipped during the “Collecting the values of local gradients” stage.

Otherwise (when the output of “8.8.3.6.6 Decision process for a luma sample” at least for one sample belonging to the “region of blocking artifact” is equal to 0) , boundary position of edge operator is skipped during the “Collecting the values of local gradients” stage.

Alternatively, when smaller edge operator size is selected (e.g. 2x2) , top-left corner window position may be removed from gradient estimation, so that only top template area and left template are processed in local gradient estimation during DIMD HoG estimation process.

Usage of multiple edge operators of different sizes and types

In some embodiments, multiple gradient operators with different sizes may be cooperatively utilized in the method shown in FIG. 8. Specifically, the intra prediction mode selection in S204 may be based on a weighted sum of gradients of the samples within the reference area obtained with the multiple gradient operators.

Although the use of larger-size edge operator might improve the accuracy of HoG because more samples are involved in estimating gradient parameters, the samples positions, which these gradient parameters are estimated for, are more distant from a block as shown in FIG. 20. That can result in losing the spatial correlation between the blocks, for which we derive intra prediction modes, and the samples involved by larger-size edge operators (e.g., of 5x5 size) in the calculation of gradient parameters.

To avoid this drawback, we could use multiple edge operators of different sizes and types as shown in FIGs. 21 to 24. In these embodiments, gradient parameters are collected using the edge operators of different size so that we get the information on gradient parameters corresponding to various distances from block boundaries. Therefore, we can take advantage of this information by making it more consistent by means of averaging or filtering out the gradient parameters obtained from the edge operators of different sizes.

The averaging of the gradient parameters obtained from the edge operators of different sizes undergoes the process of building HoGs for each edge operator used so that we get the number of HoGs equal to the number of the edge operators used. Eventually, we get the integral values |G|_Σ (Θ) of each HoG bin corresponding to the Θ angle as the result of averaging the HoGs for each edge operator used:

where w_i (Θ) and |G|_i (Θ) are the weight and the gradient magnitude of the HoG bin corresponding to the Θ angle taken from the HoG corresponding to the ith edge operator, respectively.

Note that the w_i (Θ) weight can be taken from a look-up table that contains predefined values of these weights, or obtained by on-line training using different methods (e.g., optimization techniques, which minimizes the template matching cost, or more advanced machine learning methods) .

In some embodiments, the operation of estimating horizontal gradients and vertical gradients of samples within the reference area with the gradient operator may include, for a prediction direction: calculating a plurality of gradient magnitudes corresponding to a plurality of operators respectively; selecting one of the plurality of operators; and selecting a gradient magnitude corresponding to the selected one of the plurality of operators as magnitude value of the prediction direction.

One example of these embodiments comprises a filtering process based on median filter. At step 1, we collect gradient magnitudes |G|_i (Θ) of the HoG bin corresponding to the angle Θ taken from the HoG corresponding to the ith edge operator and find the median values of this gradient magnitude set and index k (Θ) of the edge operator. The found median value of the gradient magnitudes |G|_i (Θ) is associated with:

Note that besides median function, another functions could be used, including but not limited to: argmin (·) , argmax (·) , argmin (|mean (|G|_i (Θ) ) -|G|_i (Θ) |) , etc. In this way, gradient magnitude obtained based on the operator with the index k may be selected for the prediction direction Θ in the HoG. It should be noticed, the gradient magnitudes of different operators |G|_i (Θ) may be normalized before they are compared with each other. Normalization could be performed by calculating the least common multiple of the filter norms of the filter kernels used by different edge operators and multiplying the gradient magnitudes by a normalization factor which is a ratio of the least common multiple to the norm of the corresponding norm of the filter kernel. The norm of a filter kernel could be obtained by summing up the positive coefficients of the filter kernel.

In another example, the value of index k of the edge operator is determined by a value of variance:
k (Θ) =argmin (σ² (R_i) )
arg (G_k (R_i) ) =Θ

where R_i is a set of samples that are used to estimate the edge operator value at position i, given that an angle determined at position i using the gradient G_k estimated using an edge operator with an index k is equal to Θ.

In this section, the function arg (·) returns an angle of the input gradient G, which is a pair of horizontal and vertical gradients, i.e. G_x and G_y, respectively. Function arg (G) , for example, may be defined as

Note that other functions could be used, to determine the index k, including but not limited to: k=argmax (σ² (R_i) ) , k=argmean (σ² (R_i) ) , k=argmedian (σ² (R_i) ) , etc.. These alternative functions are used to determine the value of k with the same condition defined for the angle calculated in position i: arg (G_k (R_i) ) =Θ. At step 2, we use the |G|_k (Θ) as the magnitude value of the integral HoG produced as a result of multiple edge operators.

Non-adjacent DIMD

In some embodiments, the reference area of the current block may include one or more reference lines. Moreover, the reference lines may be non-adjacent to the current block. Specifically, the reference area may include the reference lines and adjacent area of the reference lines. For example, the adjacent area of a reference line may refer to both the left and right areas to the reference line; both the above and below areas of the reference line; either the left or right area to the reference line; or, either the above or below area of the reference line. The operation S205 of updating the list of candidate prediction modes for the current block as shown in FIG. 8 may include: updating the list of candidate prediction modes for each of the one or more reference lines. The method corresponding to these embodiments may be referred to as non-adjacent DIMD.

Derivation of the non-adjacent DIMD modes list for a reference line could be performed as follows:

● determine a size of a DIMD operator based on reference line and a block size

● obtain a HoG using the DIMD operator applied for a template area specified for the reference line (see FIGs. 25 and 26)

● Sort the HoG by magnitude value

● Select the first N modes from the sorted HoG (e.g., N=5)

These steps are performed independently per each reference line, so that a different list of non-adjacent DIMD modes is obtained per each reference line.

In an alternative embodiment, this process is not invoked for reference line 1, because the template area of for this line corresponds to the casual DIMD process.

FIGs. 25 and 26A illustrate the design of this technique that we will refer to as “non-adjacent DIMD” . In FIG. 25, DIMD with the edge operator of 3x3 size is applied to reference line 5. The distinctive feature of the design shown in FIG. 26A compared to FIG. 25 is in the usage of 5x5 edge operator instead of 3x3 edge operator.

It is worth noting that gradient operators of different sizes and types as well as different angular mode ranges can be assigned to different Reference lines. Table 11 illustrates various options of gradient operator sizes and types, angular mode ranges as well their combinations.

Table 11. Exemplary dependencies of the gradient operator parameters and angular mode ranges on non-adjacent reference lines

In addition, it is worth noting that reference area associated with non-adjacent reference lines can have position different from the positions shown in Figure 25 and Figure 26A. Figure 26B and Figure 26C depict reference area positions alternative to the one shown in Figure 25 and Figure 26A. Other positions are also possible.

Non-adjacent DIMD is aimed at deriving angular modes by estimating gradients and, eventually, directionality within the reference area pointed out by the reference line. This mechanism is applicable not just to one tool but a wide range of video coding techniques such as TMRL (Template-based Multiple Reference Line intra prediction) , MPM with the explicit indication of a reference line, or even a standalone mechanism. In the case of MPM and TMRL, i.e. when both an intra prediction mode and a reference line index are signaled as shown in FIG. 27, we add the modes derived by non-adjacent DIMD as additional options to either an MPM list or a TMRL one. As FIG. 27 illustrates, before parsing the intra prediction mode index in the MPM list, we obtain the reference line index. Thus, we can add the derivation results of non-adjacent DIMD for the obtained reference line as one or more options to the MPM list. Please, note that the derivation result of non-adjacent DIMD can include more than one intra prediction mode. In addition, for each directional mode derived by non-adjacent DIMD, we can add a range of angular modes (e.g., ±2 angular modes with regard to a directional mode derived by non-adjacent DIMD) . It is worth noting that the farther the reference line is located from the current block, the wider this range of angular modes can be due to the inaccuracy of the DIMD mechanism. Similarly, we can add pairs of intra prediction modes derived by non-adjacent DIMD for each non-adjacent reference line and the reference line indices to a TMRL list. All the above –mentioned notes regarding the intra prediction modes derived by non-adjacent DIMD are also applicable to TMRL.

Besides, non-adjacent DIMD can be used as a standalone intra-prediction method as shown in FIG. 28. In this case, we could signal the intra_non_adj_dimd_flag flag, which indicates whether non-adjacent DIMD is enabled or not. If the value of the intra_non_adj_dimd_flag flag equals 1, we parse the value of the intra_non_adj_dimd_idx index, which indicates the index of an intra prediction parameter set within the non-adjacent DIMD list. Thus, each entry of the non-adjacent DIMD list could represent a pair of: 1) an reference line index and 2) an intra-prediction mode or a combination of intra-prediction modes derived by non-adjacent DIMD for the given reference line indicated by the reference line index.

An example of applying Non-adjacent DIMD for a TMRL candidate list is illustrated in FIG. 29. In the current TMRL design, a list of 10 intra prediction modes is specified for each of the available reference line when a candidate list is prepared. In this solution it is proposed to prepare a TMRL candidate list using intra prediction modes that are determined in DIMD process that is invoked for a template area specified for a non-adjacent reference line. The steps of this solution is as follows: 1) for each reference line specified in TMRL list derivation process, determine a list of non-adjacent DIMD-modes; and 2) add non-adjacent DIMD candidates to the TMRL list prior the TMRL sorting, wherein each non-adjacent DIMD candidate is composed of the corresponding reference line and a mode from the list of non-adjacent DIMD-modes that is constructed for that reference line.

Usage of machine learning to determine parameters of edge operators

In some embodiments, the size and/or coefficients of the gradient operator may be determined via a machine learning model.

In this embodiment, parameters of edge operators (such as their filter coefficients) and parameters, which define the selection process of edge operators of different sizes and types, are derived and adjusted by using the methods of machine learning as shown in FIG. 30. DIMD input data such as template samples is sent to the module that implements DIMD mechanism and in parallel to the modules used to perform on-line training to derive filter coefficients of edge operators and to adjust the parameters of the edge operator selection process. These modules used for on-line training also receive DIMD output data such as encoder-side decisions (e.g., SAD between an original block and a DIMD based predictor) , which allow these modules to estimate the correctness of the decisions made by the DIMD algorithm. Using machine leaning methods (e.g., convolutional neural networks) , these modules might

● update the filter coefficients of edge operators available in DIMD and extend the set of available edge operators; and/or

● adjust parameters (e.g., threshold values such as block sizes, activity values within block templates) of the edge operator selection process to increase the rate of correct decisions.

The on-line training to adjust the selection mechanism of edge operators is decoupled from the on-line training to derive filter coefficients for edge operators. This decoupling is necessary to keep the implementation training complexity relatively reasonable. Results of the “the on-line training to adjust the selection mechanism of edge operators” are used as a setup parameters for “the on-line training to derive filter coefficients for edge operators” .

The stages of the on-line training could be defined as follows:

● In stage 1, when previous training data is not available, default edge operators and filter coefficients may be used when performing encoding/decoding operations of a picture.

● In stage 2, results of stage 1 are used to perform the on-line training processes of DIMD shown in FIG. 30.

● In stage 3, parameters of DIMD edge operators determined in stage 2 are applied to encode (at the encoder side) or decode (at the decoder side) a next frame.

This training procedure could be beneficial when coding a video sequence using Random Access configuration (see FIG. 31) , wherein several temporal layers are defined.

In this scenario, DIMD operator parameters could be obtained by an on-line training from a frame of lower Temporal Layer ID (e.g. Temporal Layer ID #0) , and these parameters could be applied to a frame of a higher Temporal Layer ID (e.g. Temporal Layer ID #2) . The beneficial effect in this case is that the training procedure may be performed in parallel with the encoding (or decoding) of temporal layers, thus reducing decoding structural delay.

Signaling size/type of edge operator

When the intra-prediction method as described in any of the foregoing embodiments is applied in a video decoder, an indication may be received from an encoder, which is configured to indicate the size or type of the gradient operator. The size or type of the edge operator could be indicated by an encoder and parsed by a decoder per block, per slice, per picture, or per sequence. The indication may indicate directly the size or type of the gradient operator; alternatively, the indication may indicate a relation between the size or type of the gradient operator and a size of the reference area.

For signaling within a slice header structure the following exemplary syntax could be used (see Table 12) . In Tables 12 to 15 denotations used in the specification of H. 266 /Versatile Video Coding standard is used.

Table 12. Exemplary syntax for edge operator signaling

In Table 11, the value of sh_dimd_edge_op_flag indicates the presence of signaling of the alternative edge operator.

The value of sh_dimd_edge_op_type indicates the type and size of the edge operator being used. Specifically, the following codewords shown in Table 13 could be specified for sh_dimd_edge_op_type:

Table 13. Exemplary codewords for sh_dimd_edge_op_type

In an alternative embodiment, and additional signaling at slice header level could be specified to indicate the block sizes to which the alternative edge operators should be applied (see Table 14) :

Table 14. Exemplary syntax for edge operator signaling with indication of the block sizes to which the alternative edge operators should be applied

Exemplary codewords for block size threshold indication (sh_dimd_edge_bs_idx_thr) are given in Table 15 below.

Table 15. Exemplary codewords for sh_dimd_edge_op_type

Note, that similar signaling that is descried in Table 12 to 15 could be applied at picture header, sequence parameter set or picture parameter set.

For application with signaling at block level a flag could be indicated that is used by an encoder and decoder to specify that an alternative edge operator should be applied in HoG derivation for the template area of the block.

Filtering of template/reference area before applying edge operators

The template (reference) area where edge operators are applied to can be filtered (e.g., for suppressing sensor or quantization noise) before applying edge operators to it. This embodiment discloses the design where spatial filters are used to perform this filtering although other types of filters (e.g., frequency-domain filters, wavelet-based filters) might be applicable as well. FIG. 32 and 33 illustrate an example of the filtering process when a regular spatial filter of 3x3 size is used:

Note that filters of other sizes (e.g., 5x5) are applicable as well. The selection of filters depends on noise level in a picture and other reasons. Anyway, this selection is not limited in the present disclosure.

Since the template (reference) area has limited size, the usage of a regular filter is not always possible near the borders of the template (reference) area due to the lack of available samples there. In FIG. 32, the filtering near the template area border is performed using irregular filters of different sizes (2x2, 2x3, 3x2, and 8-tap filters) as shown below:

In order to avoid handling all these corner cases, padding depicted in FIG. 33 can be used to fill unavailable samples with some values. The padding can be performed by copying available samples and putting them on the places of unavailable ones. Another ways to pad unavailable samples is extrapolation when they are predicted using neighboring available samples.

This filtering can be turned on unconditionally or subject to different conditions. For example, the conditions of enabling this filtering process can be aligned with the conditions of applying 5x5 edge operators.

FIG. 34 illustrates a flowchart of steps of obtaining values of Gx and Gy for building a histogram of gradients. The basic idea behind this solution is to determine the parameters of the edge operators used in DIMD to obtain values of Gx and Gy, subject to the parameters of a current block, the availability of its neighboring reconstructed samples, and other features of these neighboring reconstructed sample. All the proposed changes aim at increasing the accuracy of the derived intra prediction modes and their parameters (e.g., weights for fusion) that results in improving the compression efficiency of DIMD.

The steps shown in FIG. 34 may be identical at both encoder and decoder sides.

● Input block parameters 1101 comprise, at least, the location of a current block and its size. This information is used to check the availability of neighboring reconstructed samples (at stage 1111) . The check of availability could be performed as described in the prior art (e.g. H. 266/VVC specification, “8.4.5.2.8. Reference sample availability marking process” ) . Since the size of edge operators to be applied to the template areas can exceed the size of the minimal block size (i.e. 4) , we determine how many neighboring reconstructed samples are available for both the top and left sides of the current block. The maximum depth of the neighboring reconstructed samples to be checked for their availability should not be greater than the maximum size of the edge operators, which can be applied to these reconstructed samples.

● Then at stage 1112, the template areas are defined based on the input block parameters (e.g., current block size) and the availability of neighboring reconstructed samples.

● At the next stage (1113) , some block parameters (such as an image activity function to evaluate the smoothness of the content near the current block) are estimated by deriving them from the available neighboring reconstructed samples.

● Based on the input block parameters and the estimated block parameters derived at stage 1113, the parameters (i.e. size and type) of an edge operator is determined (1114) .

● At stage 1121, the selected edge operator and the set of the selected edge operators are applied to the template areas defined at stage 1112 to build a histogram of gradients (HoG) , like it is done in the prior art.

Most of the state-of-the-art hybrid video codecs perform an intra prediction operation for a block of a picture, and the subdivision of a picture into blocks is a part of state-of-the art signaling. Some embodiment of the present disclosure utilizes information about the size of the block just like many other size-dependent tools, e.g. Mode Dependent Intra Smoothing (MDIS) or Position Dependent Prediction Combination (PDPC) . Hence, it does not require additional block-level signaling if applied in the encoding and decoding processes of state-of-the-art video coding standards, such as H. 266/VVC, ECM, and AV1.

The most important advantage of this approach is that we can avoid additional RD checks and thus not to increase encoder-side run-time. Beneficial effect of the proposed invention could be achieved by reducing edge operator size to achieve better spatial correlation of DIMD template with smaller blocks. For example, a 2x2 edge operator could be used instead of 3x3 Sobel edge operator that is currently used in HoG construction design. In this case, a worst-case complexity could be reduced, because the number of operations for smallest intra-coded blocks is reduced in potentially the longest data processing path.

In addition, smaller filter sizes of an edge operator could give more accurate gradients estimation for a coded block, because the area where these gradients are estimated becomes spatially closer to the current block. When a filter core of an edge operator is convolved with some samples of a template area, an estimation of a gradient is obtained for the central spatial position of this area. It could be noticed, that each dimension of this area (i.e., width and height) should be equal or larger than corresponding dimension of the filter’s core. Therefore, smaller size of a filter core may require smaller template area size and hence, provides closer position of its center to the current block.

Additional beneficial effect of the invention is a more accurate estimation of a HoG for blocks of larger size by engaging more template samples.

Other potential benefits of some embodiments of the present disclosure may include:

● Reduction of bitrate: more accurate prediction leads to smaller residuals, and therefore less numbers of bits required to be transferred from an encoder to a decoder.

● Improvement of picture quality: the invention improves the accuracy of prediction and thus prediction provides better PSNR as compared with the prior art.

FIG. 35 conceptually illustrates an apparatus 300 with which some embodiments of the invention are implemented. The apparatus 300 may be an encoding device or a decoding device. The apparatus 300 may be a computer (e.g., a desktop computer, personal computer, tablet computer, etc. ) , phone, PDA, or any other sort of electronic device. Such an apparatus includes various types of computer readable media and interfaces for various other types of computer readable media. The apparatus 300 includes a processor 302 and a memory 304. The memory 304 is configured to store executable instructions that, when executed by the processor, cause the processor to perform any one of the foregoing decoding or encoding methods, or the intra-prediction method.

The processor 302 may be a single processor or a multi-core processor in different embodiments. In some embodiments, the processor may include a GPU, NPU or DSP which may offload various computations or complement the image processing provided by the processor 302.

Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media) . Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM) , recordable compact discs (CD-R) , rewritable compact discs (CD-RW) , read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM) , a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc. ) , flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc. ) , magnetic and/or solid state hard drives, read-only and recordable discs, ultra density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) . In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself. In addition, some embodiments execute software stored in programmable logic devices (PLDs) , ROM, or RAM devices.

As used in this specification and any claims of this application, the terms “computer” , “server” , “processor” , and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. As used in this specification and any claims of this application, the terms “computer readable medium, ” “computer readable media, ” and “machine readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.

The present disclosure further provides a computer readable media which is configured to store executable instructions. When the instructions are executed by a processor, the processor may perform any one of the foregoing methods and processes. Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium) . When these instructions are executed by one or more computational or processing unit (s) (e.g., one or more processors, cores of processors, or other processing units) , they cause the processing unit (s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard drives, erasable programmable read only memories (EPROMs) , electrically erasable programmable read-only memories (EEPROMs) , etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.

In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

While the disclosure has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. In addition, a number of the figures conceptually illustrate processes and methods. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, the process could be implemented using several sub-processes, or as part of a larger macro process.

The foregoing is merely embodiments of the present disclosure, and is not intended to limit the scope of the disclosure. Any transformation of equivalent structure or equivalent process which uses the specification and the accompanying drawings of the present disclosure, or directly or indirectly application in other related technical fields, are likewise included within the scope of the protection of the present disclosure.

Claims

A intra-prediction method for video coding, comprising:

acquiring a reference area of a current block;

determining a size or a type of a gradient operator;

estimating horizontal gradients and vertical gradients of samples within the reference area with the gradient operator; and

selecting an intra prediction mode or updating a list of candidate prediction modes for the current block based on the horizontal gradients and the vertical gradients.
The method of claim 1,

wherein the size of the gradient operator is determined based on a size of the current block.
The method of claim 2, wherein when the size of the current block is less than a first threshold, the size of the gradient operator is determined as 2 by 2.
The method of claim 3, wherein the 2 by 2 gradient operator comprises a horizontal operator Fx and a vertical operator Fy, and:
The method of claim 2, wherein when the size of the current block is larger than a second threshold, the size of the gradient operator is determined as 5 by 5.
The method of claim 5, wherein the 5 by 5 gradient operator comprises a horizontal operator Fx and a vertical operator Fy, and:
The method of claim 1,

wherein the gradient operator comprises a first operator for a left part of the reference area and a second operator for a top part of the reference area;

wherein a size of the first operator corresponds to a number of reference lines in the left part of the reference area, and a size of the second operator corresponds to a number of reference lines in the top part of the reference area.
The method of claim 7,

wherein the size of the first operator is different from the size of the second operator;

wherein the first operator and the second operator are normalized.
The method of claim 1, wherein the determining the type of the gradient operator comprises: determining a filter core of the gradient operator.
The method of claim 9, wherein the filter core comprises a Prewitt operator or a Scharr operator.
The method of claim 10, further comprising:

determining smoothness of the reference area;

wherein the determining the filter core of the gradient operator comprises:

determining the filter core as the Scharr operator responsive to the reference area being smooth; and

determining the filter core as the Prewitt operator responsive to the reference area being non-smooth.
The method of claim 9, wherein the size and the filter core of the gradient operator is determined based on at least one selected from:

normalized horizontal and vertical gradients of samples within the reference area which are acquired with a predefined operator; and

a statistical moment of samples within the reference area.
The method of claim 1, further comprising:

determining positions of boundaries between different coding units; and

excluding points that intersect the boundaries from the samples within the reference area.
The method of claim 1,

wherein the gradient operator comprises a plurality of operators with different sizes.
The method of claim 14,

wherein the intra prediction mode is selected based on a weighted sum of gradients of the samples within the reference area obtained with the plurality of operators.
The method of claim 14,

wherein the estimating horizontal gradients and vertical gradients of samples within the reference area with the gradient operator comprises, for a prediction direction:

calculating a plurality of gradient magnitudes corresponding to the plurality of operators respectively;

selecting one of the plurality of operators; and

selecting a gradient magnitude corresponding to the selected one of the plurality of operators as magnitude value of the prediction direction.
The method of claim 16,

wherein among the plurality of operators, the selected one of the plurality of operators has:

a median gradient magnitude;

a maximal gradient magnitude;

a minimal gradient magnitude; or

a gradient magnitude that is closest to an average value of gradient magnitudes of the plurality of operators.
The method of claim 16,

wherein among the plurality of operators, the selected one of the plurality of operators has a minimal variance of samples which are used for estimating gradients.
The method of claim 1,

wherein the reference area of the current block comprises one or more reference lines.
The method of claim 19,

wherein the reference area of the current block further comprises adjacent area of the one or more reference lines.
The method of claim 19,

wherein the updating the list of candidate prediction modes for the current block based on the horizontal gradients and the vertical gradients comprises:

updating the list of candidate prediction modes for each of the one or more reference lines.
The method of claim 19,

wherein the one or more reference lines are non-adjacent to the current block.
The method of claim 22,

wherein the list of candidate prediction modes is a Template-based Multi-Reference Line Intra prediction (TMRL) list;

wherein the reference area of the current block comprises a plurality of reference lines;

wherein the estimating horizontal gradients and vertical gradients of samples within the reference area with the gradient operator comprises: estimating horizontal gradients and vertical gradients of samples within each of the plurality of reference lines;

wherein the updating the list of candidate prediction modes comprises: adding candidates for the plurality of reference lines based on the horizontal gradients and vertical gradients of samples within each of the plurality of reference lines.
The method of claim 1, wherein the determining the size of the gradient operator comprises:

determining the size and/or coefficients of the gradient operator via a machine learning model.
The method of claim 24, wherein samples within a template area and corresponding encoder-side decisions on the size and/or coefficients of the gradient operator are adopted for training the machine learning model.
The method of claim 1, wherein the determining the size of the gradient operator comprises:

receiving an indication indicating the size or type of the gradient operator.
The method of claim 26, wherein

the indication indicates directly the size or type of the gradient operator; or

the indication indicates a relation between the size or type of the gradient operator and a size of the reference area.
The method of claim 26, wherein the indication is included in at least one selected from: a slice header, a picture header, a sequence parameter set, and a picture parameter set.
The method of claim 1, further comprising:

before the estimating horizontal gradients and vertical gradients of samples within the reference area with the gradient operator:

applying a specific filter to update the reference area.
The method of claim 29, wherein the specific filter comprises at least one selected from:

a spatial filter, a frequency-domain filter, and a wavelet-based filter.
The method of claim 29,

wherein the specific filter comprises one or more irregular filters;

wherein the one or more irregular filters are applied for one or more borders of the reference area.
A video coding apparatus, comprising a processor and a memory, wherein the memory is configured to store executable instructions that, when executed by the processor, cause the processor to perform the method of any of claims 1 to 31.
A computer readable medium storing executable instructions that, when executed by a processor, cause the processor to perform the method of any of claims 1 to 31.