TW202412522A - Constraining convolution model coefficient - Google Patents

Abstract

A method for performing cross component prediction by constraining the coefficients of a component prediction model is provided. A video coder receives data for a block of pixels to be encoded or decoded as a current block of a current picture of a video. The video coder derives a set of coefficients based on corresponding input and output component samples. The video coder constrains the derived set of coefficients based on a set of constraints. The video coder applies the constrained set of coefficients as a component prediction model to generate a predictor for the current block. The video coder encodes or decodes the current block by using the generated predictor.

Description

Coefficients of the Constrained Convolution Model

The present application relates to video coding and decoding. In particular, the present application relates to a method for coding and decoding a picture element block by cross-component prediction.

Unless otherwise indicated herein, the methods described in this section are not prior art to the claims listed below and are not admitted to be prior art by inclusion in this section.

High Efficiency Video Coding (HEVC) is an international video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on a hybrid block-based motion-compensated DCT-like transform coding architecture. The basic unit of compression, called a coding unit (CU), is a 2Nx2N square block, and each CU can be recursively divided into four smaller CUs until a predetermined minimum size is reached. Each CU contains one or more prediction units (PUs).

Versatile Video Coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Group (JVET) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. The input video signal is predicted from the reconstruction signal, which is derived from the coded picture region. The predicted residual signal is processed by a block transform. The transform coefficients are quantized and entropy coded together with other side information in the bitstream. The reconstructed signal is generated by the prediction signal and the reconstruction residual signal after inverse transforming the dequantized transform coefficients. The reconstructed signal is further processed by in-loop filtering to remove coding artifacts. The decoded pictures are stored in a frame buffer and used to predict future pictures in the input video signal.

In VVC, coded pictures are divided into non-overlapping square block areas, represented by related coding tree units (CTUs). The leaf nodes of the coding tree correspond to coding units (CUs). Coded pictures can be represented by a set of slices, each slice includes an integer number of CTUs. Each CTU in a slice is processed continuously by raster scanning. Bi-prediction (B) slices can be decoded using intra-frame prediction or inter-frame prediction, using up to two motion vectors and reference indices to predict the sample values of each block. Prediction (P) slices are decoded using intra-frame prediction or inter-frame prediction, using up to one motion vector and reference index to predict the sample values of each block. Intra-frame (I) slices are decoded using intra-frame prediction only.

A CTU can be partitioned into one or more non-overlapping coding units (CUs) using a quadtree (QT) with a nested multi-type tree (MTT) structure to accommodate various local motion and texture characteristics. A CU can be further partitioned into smaller CUs using one of five partitioning types: quadtree partitioning, vertical binary tree partitioning, horizontal binary tree partitioning, vertical center-side tritree partitioning, and horizontal center-side tritree partitioning.

Each CU contains one or more prediction units (PU). The prediction unit, together with the associated CU syntax, works as a basic unit of signal predictor information. The specified prediction process is used to predict the value of the associated primitive sample within the PU. Each CU can contain one or more transform units (TU) to represent the prediction residual block. The transform unit (TU) consists of a transform block (TB) of luminance samples and two corresponding transform blocks of chrominance samples, each TB corresponding to a sample residual block of a color component. Integer transforms are applied to the transform blocks. The level values of the quantization coefficients are entropy encoded in the bitstream along with other side information. The terms coding tree block (CTB), coding block (CB), prediction block (PB), and transform block (TB) are defined to specify the two-dimensional sample arrays of monochrome components associated with CTU, CU, PU, and TU, respectively. Thus, a CTU consists of one luma CTB, two chroma CTBs, and related syntax elements. Similar relationships apply to CU, PU, and TU.

For each inter-frame predicted CU, motion parameters consisting of motion vector, reference picture index, reference picture list usage index, and additional information are used for inter-frame predicted sample generation. The motion parameters can be signaled explicitly or implicitly. When the CU is encoded in skip mode, the CU is associated with one PU and there are no significant residual coefficients, no encoded motion vector increments or reference picture indices. A merge mode is specified, in which the motion parameters of the current CU are obtained from neighboring CUs, including spatial and temporal candidates and additional schedules introduced in VVC. This merge mode can be applied to any inter-frame predicted CU. An alternative to this merge mode is explicit transmission of motion parameters, where the motion vectors, the corresponding reference picture index for each reference picture list, the reference picture list usage flag, and other required information are explicitly signaled in each CU.

The following summary is illustrative only and is not intended to be binding in any way. That is, the following disclosure is provided to introduce the concepts, highlights, benefits, and advantages of the novel and non-obvious technologies described herein. Selected and not all implementations are further described in the detailed description below. Therefore, the following disclosure is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

Some embodiments of the present application provide a method for performing cross-component prediction by constraining the coefficients of a component prediction model. A video codec receives picture element block data to be encoded or decoded as a current block of a current video picture. The video codec derives a coefficient set based on corresponding input and output component samples. The video codec constrains the derived coefficient set according to a set of constraint conditions. The video codec uses the constrained coefficient set as a component prediction model to generate a predictor for the current block. The video codec uses the generated predictor to encode or decode the current block.

In some embodiments, the component prediction model is a cross-component model that generates predicted chrominance samples based on reconstructed luma samples of the current block. The corresponding input and output component samples are corresponding luma and chrominance samples of a sample region adjacent to the current block. The predictor may include the generated predicted chrominance samples. In some embodiments, the component prediction model is a convolution model; the set of coefficients is derived by solving a matrix equation between the corresponding input and output component samples.

In some embodiments, the encoder constrains the derived coefficient set by clipping coefficients at a clipping threshold, clipping different coefficients at different clipping thresholds, or limiting coefficients to a predefined range. In some embodiments, the derived coefficient set and the constrained coefficient set are represented in the encoder by a fixed point, whose fractional part includes N bits, and the size of the predefined range is magnified based on 1<<N. In some embodiments, when the derived coefficient exceeds the predefined range, the coefficient set is set to be equal to the identification filter, or the derived coefficient set is not used to encode or decode the current block (disabling CCCM mode).

In the following detailed description, many specific details are explained by way of example to provide a comprehensive understanding of the relevant teachings. Any variation, derivation, and/or extension based on the teachings described herein are within the scope of protection of this application. In some cases, well-known methods, procedures, components, and/or circuits related to one or more exemplary embodiments disclosed herein may be described at a relatively high level without detailed description to avoid unnecessarily covering up various aspects of the teachings disclosed herein. I. Cross-Component Linear Model ( CCLM )

The Cross-Component Linear Model (CCLM) or Linear Model (LM) mode is a cross-component prediction mode in which the chrominance components of a block are predicted from the collocated reconstructed luminance (Luma) samples by a linear model. The parameters of the linear model (e.g., scale and offset) are derived from the already reconstructed luminance and chrominance (Chroma) samples adjacent to the block. For example, in VVC, the CCLM mode exploits the inter-channel dependencies to predict chrominance samples from reconstructed luminance samples. This prediction is performed using a linear model of the following form: (1)

In formula (1) Represents the predicted chroma sample in the CU (or the predicted chroma sample of the current CU), Represents the down-sampled reconstructed luma samples of the same CU (or the corresponding reconstructed luma samples of the current CU).

The CCLM model parameters α (scaling parameter) and β (offset parameter) are derived from up to four adjacent chrominance samples and their corresponding downsampled luma samples. In LM_A mode (also denoted as LM-T mode), only the upper or top adjacent samples are used to calculate the coefficients of the linear model. In LM_L mode (also denoted as LM-L mode), only the left samples are used to calculate the coefficients of the linear model. In LM-LA mode (also denoted as LM-LT mode), both the left and top samples are used to calculate the coefficients of the linear model.

FIG. 1 conceptually illustrates the chrominance and luma samples used to derive linear model parameters. The figure shows a current block 100 with luma component samples and chroma component samples in a 4:2:0 format. The luma and chroma samples adjacent to the current block are reconstructed samples. These reconstructed samples are used to derive the linear model (parameters α and β) of the cross components. Since the current block is in a 4:2:0 format, the luma samples are downsampled before being used in linear model derivation. In this example, there are 16 pairs of reconstructed luma (downsampled) and chroma samples adjacent to the current block. These 16 pairs of luma and chroma values are used to derive linear model parameters.

Assuming the current chroma block size is W×H, then W' and H' are set as follows: - When LM-LT mode is applied, W'=W, H'=H; - When LM-T mode is applied, W'=W+H; - When LM-L mode is applied, H'=H+W.

The upper adjacent positions are represented as S[0, -1]...S[W' -1, -1], and the left adjacent positions are represented as S[-1, 0]...S[-1, H' -1]. Then select four samples: - When the LM mode is applied (both the upper and left adjacent samples are available), S[W'/4, -1], S[3*W'/4, -1], S[-1, H'/4], S[-1, 3*H'/4]; - When the LM-T mode is applied (only the upper adjacent samples are available), S[W'/8, -1], S[3*W'/8, -1], S[5*W'/8, -1], S[7*W'/8, -1]; - When the LM-L mode is applied (only the left adjacent samples are available), S[-1, H'/8], S[-1, 3*H'/8], S[-1, 5*H'/8], S[-1, 7*H'/8];

The four adjacent luminance samples at _the selected position are downsampled and compared four times to find the two larger _{values: x0A and x1A, and the two smaller values: x0B and x1B. Their corresponding chrominance sample values are denoted as y0A, y1A} , y0B , and _{y1B . Then ,} XA _{, XB} , _YA , and _YB are derived as follows: _Xa = ( _x0A + x1A +1) _>> 1 ; Xb ₌ ( x0B + x1B ₊₁ )>> ₁ (2 ) Ya = (y0A + y1A +1)>>1; _{Yb =} ( _y0B + _{y1B +1 ) >>} 1 ( ₃ )

The linear model parameters α and β are obtained according to the following formula: (4) (5)

The operation of calculating the α and β parameters according to formulas (4) and (5) can be implemented by looking up a data table. In some embodiments, in order to reduce the memory required to store the data table lookup, the diff value (the difference between the maximum and minimum values) and the parameter α are represented by exponential symbols. For example, diff is approximated by a 4-bit significant part and an exponent. Therefore, the table of 1/diff is simplified to 16 elements for 16 valid values, as shown below: DivTable[] = {0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0} (6)

This reduces the complexity of the calculations and the memory size required to store the required tables.

In some embodiments, in order to obtain more samples for calculating the CCLM model parameters α and β, for the LM-T mode, the upper sample is expanded to include (W+H) samples, and for the LM-L mode, the left sample is expanded to include (H+W) samples. For the LM-LT mode, both the expanded left sample and the expanded upper sample are used to calculate the coefficients of the linear model.

To match the chroma sample positions of a 4:2:0 video sequence, two types of downsampling filters are applied to luma samples to achieve a 2:1 downsampling ratio in both horizontal and vertical directions. The choice of downsampling filter is specified by a flag in the sequence parameter set (SPS) level. The two downsampling filters are as follows, corresponding to the contents of "type 0" and "type 2"("type-0" and "type-2"), respectively. (7) (8)

In some embodiments, the α and β parameter calculations are performed as part of the decoding process, not just as an encoder search operation. Therefore, no syntax is used to communicate the values of α and β to the decoder.

For chroma intra-frame mode encoding and decoding, a total of 8 intra-frame modes are allowed. These modes include five traditional intra-frame modes and three cross-component linear modes (LM_LA, LM_A and LM_L). Chroma intra-frame mode encoding and decoding can directly depend on the intra-frame prediction mode of the corresponding luminance block. The chroma intra-frame mode signal and the corresponding luminance intra-frame prediction mode are shown in the following table: Chroma Intra-frame Prediction Mode Corresponding brightness frame prediction mode 0 50 18 1 X (0 ≤ X ≤66) 0 66 0 0 0 0 1 50 66 50 50 50 2 18 18 66 18 18 3 1 1 1 66 1 4 0 50 18 1 X 5 81 81 81 81 81 6 82 82 82 82 82 7 83 83 83 83 83

Since the independent block partitioning structure of luma and chroma components is enabled in I-picture, one chroma block may correspond to multiple luma blocks. Therefore, for the chroma derivative mode (DM) mode, the intra-frame prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited.

A unified binarization table (mapped to bin string) is used for chroma intra prediction mode according to the following table: Chroma Intra-frame Prediction Mode bin string 4 00 0 0100 1 0101 2 0110 3 0111 5 10 6 110 7 111

In this table, the first bin indicates whether it is normal mode (0) or LM mode (1). If it is LM mode, then the next bin indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next bin indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first bin of the binarization table of the corresponding intra_chroma_pred_mode can be discarded before entropy encoding and decoding. Or, in other words, the first bin is inferred to be 0 and therefore will not be encoded or decoded. This single binarization table is used for the cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two bins in the table are context encoded or decoded with their own context model, and the remaining bins are bypass encoded or decoded.

In addition, to reduce intra-frame chroma latency in dual trees, when the 64x64 luma codec tree node is not split (and the ISP is not used for 64x64 CUs) or is split with QT, the chroma CUs in the 32x32/32x16 chroma codec tree nodes are allowed to use CCLM in the following manner: If the 32x32 chroma node is not split or is split with QT, all chroma CUs in the 32x32 node can use CCLM. If the 32x32 chroma node is split with Horizontal BT and the 32x16 child node is not split or is split with Vertical BT, all chroma CUs in the 32x16 chroma node can use CCLM. Under all other luma and chroma codec tree split conditions, chroma CUs are not allowed to use CCLM. II. Multi-Mode CCLM ( MMLM)

Multi-model CCLM mode (MMLM) uses two models to predict chrominance samples from luma samples of the entire CU. Similar to CCLM, three multi-model CCLM modes (MMLM_LA, MMLM_A, and MMLM_L) are used to indicate whether to use both upper and left neighboring samples, only upper neighboring samples, or only left neighboring samples in model parameter derivation.

In MMLM, the neighboring luminance samples and neighboring chrominance samples of the current block are divided into two groups, each of which is used as a training set to derive a linear model (i.e., derive a specific α and β for a specific group). In addition, the samples of the current luminance block are also classified according to the same rules as the classification of the neighboring luminance samples.

Figure 2 shows an example of classifying neighboring samples into two groups. The threshold ( Threshold ) is calculated as the average of neighboring reconstructed brightness samples. Neighboring samples at [x,y] are classified into the first group if Rec' _L [ x , y ] ＜= Threshold , while neighboring samples at [x,y] are classified into the second group if Rec' _L [ x , y ] ＞ Threshold . Therefore, the multi-model CCLM prediction for the chrominance samples is: Pred _c [ x,y ] = α ₁ ×Rec ʹ _L [ x,y ] + β ₁ if Rec' _L [ x,y ] ≤ Threshold Pred _c [ x,y ] = α ₂ ×Rec ʹ _L [ x,y ] + β ₂ if Rec' _L [ x,y ] ＞ Threshold III. Convolutional Cross-Component Model ( CCCM )

In some embodiments, a convolution cross-component model (CCCM) is applied to improve cross-component prediction performance. For some embodiments, the convolution model has a 7-tap filter with a 5-tap plus sign-shaped spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter includes a center (C) luma sample, which matches the chroma sample to be predicted, and the neighboring samples above/north (N), below/south (S), left/west (W), and right/east (E) of the center luma sample. FIG. 3 conceptually illustrates the spatial components of the convolution filter. The nonlinear term (denoted as P) is expressed as the quadratic value of the center brightness sample C, and scaled by the sample value range of the content: P = (C*C + midVal) ＞＞ bitDepth                                                              (9)

Therefore, for 10-bit content, the nonlinear term P is calculated as: P = (C*C + 512) >> 10

The bias term (denoted B) represents a scalar offset between input and output (similar to the bias term in CCLM) and is set to the intermediate chroma value (512 for 10-bit content). The output of the filter is computed as the convolution between the filter coefficients _ci and the input value, and is clipped to the valid chroma sample range: predChromaVal = c ₀ C + c ₁ N + c ₂ S + c ₃ E + c ₄ W + c ₅ P + c ₆ B (10)

The filter coefficients c _i are calculated by minimizing the MSE between the reconstructed (or target) chrominance samples and the corresponding predicted chrominance samples in the reference region. Each predicted chrominance sample is generated from the collocated luma sample and its surrounding luma samples using the derived component prediction model (such as equation (10)). Equation (10) is a convolution model based on the center sample and the surrounding 4 samples (C, N, S, E, W). Equation (10) can be expanded to include taps of the center sample and the surrounding 8 samples (C, N, S, E, W, NE, NW, SE, SW). Equation (10) and its extended form can be called a component prediction model (because it can be used for cross-component prediction or intra-component prediction).

Figure 4 shows a reference region for deriving filter coefficients for the convolution model of the current block. The reference region includes (reference) rows of (chrominance) samples above and to the left of the current block 400. (In this example, the current block 400 is a PU). The reference region extends a PU width to the right and a PU height below the PU boundary. The reference region can be adjusted to include only available samples. The extended area of the reference region is used to support "side samples" of signed-shaped spatial filters (e.g., the N, E, W, S samples, and NW, NE, SW, SE samples described in Figure 3), which are filled in when unavailable areas.

Minimization of MSE can be achieved by calculating the autocorrelation matrix of the luma input and the cross-correlation vector between the luma input and the chroma output. The autocorrelation matrix is decomposed by LDL and the final filter coefficients are calculated using the inverse permutation method. This process is similar to the calculation of the ALF filter coefficients in ECM, however, in some embodiments, LDL decomposition is selected instead of Cholesky decomposition to avoid the use of square root operations.

In some embodiments, a high-order model may be used to predict chrominance samples instead of a linear model. The high-order model may include a k -tap space term, a nonlinear term (denoted as P ), and a bias term (denoted as B ). The high-order model may be specified as: (11)

in It's location The brightness samples reconstructed by downsampling, yes A nearby sample, , , ,and are model parameters. The high-level model formula can be used to derive model parameters between color components, or between reference samples of the current frame/picture and a reference frame/picture. III. Constraints on model coefficients

In some embodiments, after the matrix equation solving process, the final derived filter/model coefficients are used to generate the CCCM predictor. However, the derived filter coefficients may be unreasonable, for example, the model over-fits the reference sample, or the filter coefficients are too large. Some embodiments of the present application provide a method for constraining the derived model coefficients before generating the CCCM predictor.

FIG. 5 conceptually illustrates a data path 500 of a video codec that derives and constrains coefficients of a model (e.g., for CCCM). The constrained model coefficients are used to generate a predictor for the current block. As shown, the data path 500 begins with a matrix preparation module 510 that generates an autocorrelation matrix and a cross-correlation vector 515. The autocorrelation matrix is prepared based on corresponding input component samples 505 (X samples). The cross-correlation vector is prepared based on corresponding input and output component samples 505 (X samples and Y samples). The corresponding input and output component samples can be corresponding reconstructed luminance and chrominance samples of a reference template region adjacent to the current block.

The generated autocorrelation matrix and cross-correlation vector 515 are provided to a matrix equation solver module 530 to generate an optimized coefficient set 535. The coefficient constraint module 540 constrains the optimized coefficient 535 to generate a constrained final coefficient set 545. The constrained final coefficient 545 is ultimately used as a component prediction model 550 (e.g., CCLM or CCCM) to generate a predicted component sample 565 (e.g., chrominance of the current block) based on a reference component sample 560 (e.g., luminance of the current block). The generated predicted component sample 565 can be used as a CCCM predictor.

Different types of constraints may be applied to the optimized coefficients 535 to generate the constrained final coefficients 545. For example, in some embodiments, a predefined clipping threshold is used to clip the optimized coefficients 535 before generating the constrained final coefficients 545. In some embodiments, multiple clipping thresholds are predefined for the optimized coefficients 535, and a syntax indicating the selected threshold may be signaled. In some embodiments, multiple clipping thresholds are predefined for the optimized coefficients 535, and the selected threshold may be explicitly derived from neighboring reconstructed samples or side information. In some embodiments, the clipping thresholds for different coefficients may be all different or partially different.

In some embodiments, the coefficients of the component prediction model 550 are represented in fixed-point format, and the bit width of the integer part is limited to a certain value. In this example, before the coefficient constraint module 540 constrains, the fixed-point format of the optimization coefficient 535 is 48 bits for the integer part and 16 bits for the decimal part. After the constraint of the coefficient constraint module 540, in the example shown in Figure 5, the fixed-point format of the constraint coefficient 545 is 36 bits for the integer part and 16 bits for the decimal part. In other examples, the integer part of the constraint coefficient 545 can have different bit widths. In some embodiments, the bit width of the integer and/or decimal part of each coefficient can be completely different or partially different. For example, the constraint coefficient can be represented in a fixed-point format, with 36 bits for the integer part and 14 bits for the decimal part.

In some embodiments, the video codec may perform a shearing operation on the coefficients that are out of range. In some embodiments, if the coefficients are out of range, it may be inferred that the CCCM or CCLM mode is not enabled.

In some embodiments, the optimization coefficients 535 are clipped to a predefined range before generating the final predictor 565. For example, in terms of floating point precision, the range can be [-1, 1), [-2, 2], [-4, 4), or [-8, 8]. If the model coefficients are represented using fixed point precision and the number of bits in the fractional part is N bits, the predefined range will be expanded (1<<N). For example, if the supported range of floating point precision is [-8, 8) and the number of bits in the fractional part is 5 bits, the supported range will change from [-8, 8) for floating point precision to [-8*32, 8*32) for fixed point precision with a 5-bit fractional part.

In some embodiments, the predefined ranges for different model coefficients may be different. For example, the predefined range depends on the spatial position of the model coefficients. In some embodiments, the predefined range depends on the order of the corresponding inputs. In some embodiments, all coefficients use only one predefined range. In some embodiments, when a derived coefficient exceeds the predefined range, the final coefficient 545 is derived by clipping the coefficient to within the predefined range. In some embodiments, when a derived coefficient exceeds the predetermined range, the final coefficient 545 is set to zero. In some embodiments, when a derived coefficient exceeds the predetermined range, the coefficient of the CCCM mode is set to be equal to the identification filter (i.e., the CCCM mode does not apply filtering).

Any of the methods proposed above can be implemented in an encoder and/or a decoder. For example, any of the methods proposed can be implemented in an inter/intra/prediction module of an encoder and/or an inter/intra/prediction module of a decoder. Alternatively, any of the methods proposed can be implemented as a circuit coupled to an inter/intra/prediction module of an encoder and/or an inter/intra/prediction module of a decoder to provide information required by the inter/intra/prediction module. VI. Video Encoder Example

FIG6 shows an exemplary video encoder 600 that can encode a picture element block using a component prediction model. As shown, the video encoder 600 receives an input video signal from a video source 605 and encodes the signal into a bit stream 695. The video encoder 600 has a plurality of components or modules for encoding a signal from a video source 605, including at least a transform module 610, a quantization module 611, an inverse quantization module 614, an inverse transform module 615, an intra-frame prediction estimation module 620, an intra-frame prediction module 625, a motion compensation module 630, a motion estimation module 635, an intra-loop filter 645, a reconstructed picture buffer 650, an MV buffer 665, an MV prediction module 675, and some components in an entropy encoder 690. The motion compensation module 630 and the motion estimation module 635 are part of the inter-frame prediction module 640.

In some embodiments, modules 610-690 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device or electronic device. In some embodiments, modules 610-690 are modules of hardware circuits implemented by one or more integrated circuits (ICs) of an electronic device. Although modules 610-690 are illustrated as separate modules, some modules may be combined into a single module.

The video source 605 provides a raw video signal, presenting pixel data for each video frame, without compression. The subtractor 608 calculates the difference between the raw video pixel data of the video source 605 and the predicted pixel data 613 from the motion compensation module 630 or the intra-frame prediction module 625 as a prediction residue 609. The transform module 610 converts the difference (or residual pixel data or residual signal) into a transform coefficient 616 (e.g., by performing a discrete cosine transform, or DCT). The quantization module 611 quantizes the transform coefficient 616 into quantized data (or quantized coefficient) 612, which is encoded by the entropy encoder 690 into a bit stream 695.

The inverse quantization module 614 dequantizes the quantized data (or quantized coefficient) 612 to obtain a transform coefficient 616, and the inverse transform module 615 inversely transforms the transform coefficient 616 to generate a reconstruction residue 619. The reconstruction residue 619 is added to the predicted pixel data 613 to generate a reconstructed pixel data 617. In some embodiments, the reconstructed pixel data 617 is temporarily stored in a row buffer (not shown) for intra-picture prediction and spatial MV prediction. The reconstructed pixel is filtered by an intra-loop filter 645 and stored in a reconstructed picture buffer 650. In some embodiments, the reconstructed picture buffer 650 is an external storage of the video encoder 600. In some embodiments, the reconstructed picture buffer 650 is internal storage of the video encoder 600.

The intra-frame estimation module 620 performs intra-frame prediction based on the reconstructed pixel data 617 to generate intra-frame prediction data. The intra-frame prediction data is provided to the entropy encoder 690 to be encoded into the bit stream 695. The intra-frame prediction data is also used by the intra-frame prediction module 625 to generate the prediction pixel data 613.

The motion estimation module 635 performs inter-frame prediction by generating MVs to refer to the pixel data of the previous decoded frame stored in the reconstructed picture buffer 650. These MVs are provided to the motion compensation module 630 to generate predicted pixel data.

The video encoder 600 uses MV prediction to generate a predicted MV, instead of encoding the complete actual MV in the bitstream, and the difference between the MV used for motion compensation and the predicted MV is encoded as residual motion data and stored in the bitstream 695.

The MV prediction module 675 generates a predicted MV based on a reference MV generated for encoding a previous video frame, i.e., a motion compensation MV for performing motion compensation. The MV prediction module 675 retrieves the reference MV of the previous video frame from the MV buffer 665. The video encoder 600 stores the MV generated for the current video frame in the MV buffer 665 as a reference MV for generating the predicted MV.

The MV prediction module 675 uses the reference MV to create a predicted MV. The predicted MV can be calculated by spatial MV prediction or temporal MV prediction. The difference (residual motion data) between the predicted MV and the motion compensation MV (MC MV) of the current frame is encoded by the entropy encoder 690 into the bit stream 695.

The entropy encoder 690 encodes various parameters and data into a bit stream 695 by using an entropy coding technique, such as context-adjusting binary arithmetic coding (CABAC) or Huffman coding. The entropy encoder 690 encodes various header elements, flags, and quantized transform coefficients 612 and residual motion data as syntax elements into the bit stream 695. The bit stream 695 is then stored in a storage device or transmitted to a decoder via a communication medium such as a network.

The in-loop filter 645 performs filtering or smoothing operations on the reconstructed pixel data 617 to reduce encoding and decoding artifacts, especially at block boundaries. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 645 include a deblocking filter (DBF), a sample adaptive offset (SAO), and/or an adaptive loop filter (ALF).

FIG. 7 shows a portion of a video encoder 600 that derives and uses a component prediction model by constraining its coefficients. As shown, an initial predictor generation module 720 provides an initial predictor 715 to a component prediction model 710. The initial predictor 715 may include component samples (luminance or chrominance) of a reference block for predicting component samples of a current block, or component samples of the current block for cross-component prediction. The component prediction model 710 is applied to the initial predictor 715 to generate a fine predictor 725. Samples of the fine predictor 725 may be used as the prediction primitive data 613. The initial predictor 715 may be a reconstructed luminance sample of the current block, while the fine predictor 725 may be a predicted chrominance sample of the current block.

When the current block is coded by inter-frame prediction, the motion estimation module 635 provides an MV, which is used by the motion compensation module 630 to identify the reference block in the reference picture as the initial predictor 715. When the current block is coded by intra-frame prediction, the intra-frame prediction estimation module 620 provides an intra-frame mode, which is used by the intra-frame prediction module 625 to generate an intra-frame prediction of the current block as the initial predictor 715. Then, the component samples of the initial predictor 715 can be used as input to the component prediction model 710.

In order to obtain the component prediction model 710, the regression data selection module 730 retrieves the required component samples as regression data from the reconstructed image buffer 650. The regression data can be taken from the area within and/or around the current block in the current image, and the area within and/or around the reference block in the reference image. The retrieved regression data (i.e., the required component samples) include corresponding input (X) and output (Y) component samples for determining the coefficients or parameters of the component prediction model 710.

The model builder 705 uses the regression data (X and Y) to derive the coefficients of the component prediction model 710 using techniques such as elimination, iterative operation algorithms, or decomposition methods. In some embodiments, the model builder 705 applies certain constraints to the derived coefficients before providing the constrained coefficients for use as the component prediction model 710. The model builder 705 can constrain the coefficients by clipping at a threshold or limiting the coefficients to a predefined range. In some embodiments, the model builder 705 can apply different clipping thresholds to different coefficients. In some embodiments, the coefficients are represented by fixed points, the fractional part of which has N bits, and the predefined range is magnified by 1<<N. Constraining the coefficients of the component prediction model is described in Section IV ( VI ) above.

FIG. 8 conceptually illustrates a process 800 for constraining coefficients of a component prediction model. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the encoder 600 perform the process 800 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the encoder 600 performs the process 800.

The encoder (at block 810) receives data to be encoded as a current block of a current picture of a video.

The encoder (at block 820) derives a set of coefficients for a component prediction model based on corresponding input and output component samples. In some embodiments, the component prediction model is a cross-component model that generates predicted chrominance samples based on reconstructed luminance samples of the current block, and the corresponding input and output component samples are corresponding luminance and chrominance samples of a sample region adjacent to the current block. In some embodiments, the component prediction model is a convolution model that derives the set of coefficients by decomposition and inverse permutation based on an autocorrelation matrix between the corresponding input and output component samples.

The encoder (at block 830) constrains the derived coefficient set according to the set of constraint conditions. In some embodiments, the encoder constrains the derived coefficient set by clipping the coefficient at a clipping threshold, or clipping different coefficients at different clipping thresholds, or limiting the coefficients to a predefined range. In some embodiments, the derived coefficient set and the constrained coefficient set are represented in the encoder by floating point. In some embodiments, the coefficient is represented by a fixed point, and its fractional part has N bits, and the size of the predefined range is enlarged based on 1<<N. In some embodiments, when the derived coefficient exceeds the predefined range, the coefficient set is set to be equal to the identification filter, or a clipping operation is applied to the coefficients that are out of range, or the derived coefficient set is not used to encode or decode the current block (disable CCCM mode).

The encoder applies (at block 840) the constrained set of coefficients as a component prediction model to generate a predictor for the current block. The predictor may include the generated predicted chrominance samples.

The encoder (at block 850) encodes the current block by using the generated predictor to generate a prediction residual. VI. Video Decoder Example

In some embodiments, an encoder may signal (or generate) one or more syntax elements in a bitstream, such that a decoder may parse the one or more syntax elements from the bitstream.

FIG. 9 shows an example of a video decoder 900 that can use a component prediction model to decode a pixel block. As shown in the figure, the video decoder 900 is an image decoding or video decoding circuit that receives a bit stream 995 and decodes the content of the bit stream into pixel data of a video frame for display. The video decoder 900 has a plurality of components or modules for decoding the bit stream 995, including some components selected from an inverse quantization module 911, an inverse transform module 910, an intra-frame prediction module 925, a motion compensation module 930, an intra-loop filter 945, a decoded picture buffer 950, an MV buffer 965, an MV prediction module 975, and a parser 990. The motion compensation module 930 is part of the frame prediction module 940.

In some embodiments, modules 910-990 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 910-990 are modules of hardware circuits implemented by one or more ICs of an electronic device. Although modules 910-990 are illustrated as separate modules, some of these modules may be combined into a single module.

The parser 990 (or entropy decoder) receives the bit stream 995 and performs initial parsing according to the syntax defined by the video codec or image codec standard. The parsed syntax elements include various header elements, flags, and quantization data (or quantization coefficients) 912. The parser 990 parses out various syntax elements by using entropy coding and decoding techniques, such as context-adjusting binary arithmetic coding (CABAC) or Huffman coding.

The inverse quantization module 911 dequantizes the quantized data (or quantized coefficients) 912 to obtain the transform coefficients, and the inverse transform module 910 inversely transforms the transform coefficients 916 to generate a reconstructed residual signal 919. The reconstructed residual signal 919 is added to the predicted pixel data 913 from the intra-frame prediction module 925 or the motion compensation module 930 to generate decoded pixel data 917. The decoded pixel data is filtered by the in-loop filter 945 and stored in the decoded picture buffer 950. In some embodiments, the decoded picture buffer 950 is an external storage of the video decoder 900. In some embodiments, the decoded picture buffer 950 is internal storage of the video decoder 900.

The intra prediction module 925 receives intra prediction data from the bitstream 995 and generates prediction pixel data 913 from the decoded pixel data 917 stored in the decoded picture buffer 950. In some embodiments, the decoded pixel data 917 is also stored in a row buffer (not shown) for intra prediction and spatial MV prediction.

In some embodiments, the contents of the decoded picture buffer 950 are used for display. The display device 955 directly retrieves the contents of the decoded picture buffer 950 for display, or retrieves the contents of the decoded picture buffer to the display buffer. In some embodiments, the display device receives the pixel value from the decoded picture buffer 950 through pixel transmission.

The motion compensation module 930 generates predicted pixel data 913 from the decoded pixel data 917 stored in the decoded picture buffer 950 according to the motion compensation MV (MC MV). These motion compensation MVs are decoded by adding the residual motion data received from the bitstream 995 to the predicted MV received from the MV prediction module 975.

The MV prediction module 975 generates a predicted MV based on a reference MV generated for decoding a previous video frame, for example, a motion compensation MV for performing motion compensation. The MV prediction module 975 retrieves the reference MV of the previous video frame from the MV buffer 965. The video decoder 900 stores the motion compensation MV generated for decoding the current video frame in the MV buffer 965 as a reference MV for generating the predicted MV.

The in-loop filter 945 performs filtering or smoothing operations on the decoded primitive data 917 to reduce encoding and decoding artifacts, particularly at block boundaries. In some embodiments, the filtering or smoothing operations performed by the in-loop filter 945 include a deblocking filter (DBF), a sample adaptive offset (SAO), and/or an adaptive loop filter (ALF).

FIG. 10 shows a portion of a video decoder 900 that derives and uses a component prediction model by constraining its coefficients. As shown, an initial predictor generation module 1020 provides an initial predictor 1015 to a component prediction model 1010. The initial predictor 1015 may include component samples (luminance or chrominance) of a reference block for predicting component samples of a current block, or component samples of the current block for cross-component prediction. The component prediction model 1010 is applied to the initial predictor 1015 to generate a refined predictor 1025. Samples of the refined predictor 1025 may be used as prediction primitives 913. The initial predictor 1015 may be a reconstructed luminance sample of the current block, while the refined predictor 1025 may be a predicted chrominance sample of the current block.

When the current block is coded by inter-frame prediction, the entropy decoder 990 provides an MV, which is used by the motion compensation module 930 to identify the reference block in the reference picture as the initial predictor 1015. When the current block is coded by intra-frame prediction, the entropy decoder 990 provides an intra-frame mode, which is used by the intra-frame prediction module 925 to generate an intra-frame prediction of the current block as the initial predictor 1015. The component samples of the initial predictor 1015 can then be used as input to the component prediction model 1010.

In order to obtain the component prediction model 1010, the regression data selection module 1030 retrieves the required component samples as regression data from the decoded picture buffer 950. The regression data can be taken from the area within and/or around the current block in the current picture, and the area within and/or around the reference block in the reference picture. The retrieved regression data (i.e., the required component samples) include corresponding input (X) and output (Y) component samples for determining the coefficients or parameters of the component prediction model 1010.

The model builder 1005 uses the regression data (X and Y) to derive the coefficients of the component prediction model 1010 using techniques such as elimination, iterative operation algorithms, or decomposition methods. In some embodiments, the model builder 1005 applies certain constraints to the derived coefficients before providing the constrained coefficients for use as the component prediction model 1010. The model builder 1005 can constrain the coefficients by clipping at a threshold or limiting the coefficients to a predefined range. In some embodiments, the model builder 1005 can apply different clipping thresholds to different coefficients. In some embodiments, the coefficients are represented by fixed points, the fractional part of which has N bits, and the predefined range is magnified by 1<<N. Constraining the coefficients of the component prediction model is described in Section IV ( VI ) above.

FIG. 11 conceptually illustrates a process 1100 for constraining coefficients of a component prediction model. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing the decoder 900 perform the process 1100 by executing instructions stored in a computer-readable medium. In some embodiments, an electronic device implementing the decoder 900 performs the process 1100.

The decoder (at block 1110) receives data to be decoded as the current block of the current picture of the video.

The decoder derives (at block 1120) a set of coefficients for a component prediction model based on corresponding input and output component samples. In some embodiments, the component prediction model is a cross-component model that generates predicted chrominance samples based on reconstructed luminance samples of a current block, and the corresponding input and output component samples are corresponding luminance and chrominance samples of a sample region adjacent to the current block. In some embodiments, the component prediction model is a convolution model that derives the set of coefficients by decomposition and inverse permutation based on an autocorrelation matrix between the corresponding input and output component samples.

The decoder (at block 1130) constrains the derived set of coefficients according to the set of constraints. In some embodiments, the encoder constrains the derived set of coefficients by clipping the coefficients at a clipping threshold, or clipping different coefficients at different clipping thresholds, or limiting the coefficients to a predefined range. In some embodiments, the derived set of coefficients and the constrained set of coefficients are represented in the encoder using floating point. In some embodiments, the coefficients are represented by fixed point with N bits in the fractional part, and the size of the predefined range is magnified based on 1<<N. In some embodiments, when the derived coefficients are outside the predefined range, the coefficient set is set to be equal to an identification filter, or a clipping operation is applied to the coefficients outside the range, or the derived set of coefficients is not used to encode or decode the current block (CCCM mode is disabled)

The decoder applies (at block 1140) the constrained set of coefficients as a component prediction model to generate a predictor for the current block. The predictor may include the generated predicted chrominance samples.

The decoder (at block 1150) reconstructs the current block by using the generated predictor. The decoder can then provide the reconstructed current block for display as part of the reconstructed current picture. VII. Electronic System Example

Many of the features and applications described above are implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more computing or processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash memory drives, random access memory (RAM) chips, hard disks, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. Computer-readable media does not include carrier waves and electronic signals transmitted wirelessly or over wired connections.

In this specification, the term "software" is meant to include firmware residing in read-only memory or applications stored in magnetic memory, which can be read into memory for processing by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while maintaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of independent programs that jointly implement the software inventions described herein is within the scope of this application. In some embodiments, when the software program is installed on one or more electronic systems and runs, it defines one or more specific machine implementations that execute and implement the operations of the software program.

FIG. 12 conceptually illustrates an electronic system 1200 for implementing some embodiments of the present application. The electronic system 1200 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, etc.), a phone, a PDA, or any other type of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1200 includes a bus 1205, a processing unit 1210, a graphics processing unit (GPU) 1215, a system memory 1220, a network 1225, a read-only memory 1230, a permanent storage device 1235, an input device 1240, and an output device 1245.

Bus 1205 is collectively referred to as all system, peripheral, and chipset buses that communicatively connect the various internal devices of electronic system 1200. For example, bus 1205 communicatively connects processing unit 1210 with GPU 1215, read-only memory 1230, system memory 1220, and permanent storage device 1235.

From these various storage units, the processing unit 1210 retrieves instructions to be executed and data to be processed to perform the process of the present application. In different embodiments, the processing unit can be a single processor or a multi-core processor. Some instructions are passed to and executed by the GPU 1215. The GPU 1215 can offload various calculations or supplement the image processing provided by the processing unit 1210.

Read-only memory (ROM) 1230 stores static data and instructions, which are used by processing unit 1210 and other modules of the electronic system. On the other hand, permanent storage device 1235 is a read-write storage device. This device is a non-volatile storage unit that can store instructions and data even when the electronic system 1200 is turned off. Some embodiments of the present application use a large storage area device (such as a magnetic or optical disk and its corresponding disk drive) as permanent storage device 1235.

Other embodiments use unloadable storage devices (such as floppy disks, flash memory devices, etc., and their corresponding disk drives) as permanent storage devices. Like permanent storage device 1235, system memory 1220 is a read-write memory device. However, unlike storage device 1235, system memory 1220 is a volatile read-write memory, such as random access memory. System memory 1220 stores some instructions and data used by the processor during operation. In some embodiments, processes according to the content of the present application are stored in system memory 1220, permanent storage device 1235, and/or read-only memory 1230. For example, various memory units include instructions for processing multimedia clips according to some embodiments. From these different storage units, processing unit 1210 retrieves instructions to be executed and data to be processed in order to execute the processes of some embodiments.

Bus 1205 is also connected to input and output devices 1240 and 1245. Input device 1240 enables a user to communicate information and select commands to the electronic system. Input device 1240 includes an alphanumeric keyboard and pointing device (also known as a "cursor control device"), a camera (e.g., a webcam), a microphone or similar device for receiving voice commands, etc. Output device 1245 displays images generated by the electronic system or outputs data in other ways. Output device 1245 includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), as well as a speaker or similar audio output device. Some embodiments include devices such as a touch screen that serve as both an input and an output device.

Finally, as shown in FIG12 , bus 1205 also couples electronic system 1200 to a network 1225 via a network interface card (not shown). In this manner, the computer can be part of a network of computers, such as a local area network (“LAN”), a wide area network (“WAN”), or an intranet, or a network of networks, such as the Internet. Any or all of the components of electronic system 1200 may be used in conjunction with the present application.

Some embodiments include electronic components, such as microprocessors, storage devices, and memory, which store computer program instructions on a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact disc-read only (CD-ROM), compact disc-recordable (CD-R), compact disc-rewritable (CD-RW), digital versatile disc-read only (e.g., DVD-ROM, double-layer DVD-ROM), various recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable medium can store a computer program that can be executed by at least one processing unit, including a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files including high-level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

Although the above discussion refers primarily to microprocessors or multi-core processors that execute software, many of the functions and applications described above are performed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored in the circuits themselves. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices.

In this specification and any patent scope of this application, the terms "computer", "server", "processor", and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. In this specification, the term "display" or "show" refers to display on an electronic device. In this specification and any patent scope of this application, the terms "computer-readable medium", "computer-readable medium", and "machine-readable medium" are completely limited to tangible, physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transient signals.

Although the content of this application has been described with reference to many specific details, a person of ordinary skill in the art will recognize that the content of this application may be embodied in other specific forms without departing from the spirit of the content of this application. In addition, some figures (including Figures 8 and 11) conceptually illustrate the processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the process may be implemented using several sub-processes or as part of a larger macro process. Therefore, a person of ordinary skill in the art can understand that the content of this application is not limited by the aforementioned illustrative details, but is defined by the attached patent scope. Supplementary description

The subject matter described herein sometimes illustrates different elements contained in or connected to different other elements. It should be understood that the architecture described in this manner is merely an example, and in fact, many other architectures can achieve the same function. Conceptually, any arrangement of elements that achieve the same function is effectively "associated" to achieve the desired function. Therefore, any two elements here combined to achieve a specific function can be regarded as "associated" with each other to achieve the desired function, without considering the architecture or intermediate elements. Similarly, any two elements so associated can also be regarded as "operably connected" or "operably coupled" to each other to achieve the desired function, and any two elements that can be so associated can also be regarded as "operably coupled" to each other to achieve the desired function. Specific examples of operable coupling include, but are not limited to, physically mateable and/or physically interacting elements and/or wirelessly interactable and/or wirelessly interactable and/or logically interactable elements.

In addition, with respect to the use of substantially any plural and/or singular terms herein, a person skilled in the art may translate from the plural to the singular and/or from the singular to the plural as the context and/or application requires. For clarity, various singular/plural permutations and combinations may be explicitly proposed herein.

In addition, one of ordinary skill in the art will understand that, in general, the terms used herein, particularly in the appended claims, such as the subject matter of the appended claims, are generally intended as "open" terms. For example, the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "containing" should be interpreted as "containing but not limited to," and so on. One of ordinary skill in the art will further understand that if specific numbers introduced into the claims are intended, such intent will be expressly stated in the claims, and in the absence of such a statement, no such intent is made. For example, to aid understanding, the following appended claims may include the use of the introductory phrases "at least one" and "one or more" to introduce claim statements. However, the use of these phrases should not be construed as implying that a claim statement introduced by the indefinite article "a" or "an" limits any particular claim statement containing such an introduced claim statement to embodiments that contain only one such statement, even if the same claim statement includes the introductory phrase "one or more" or "at least one" and an indefinite article such as "an" or "an." For example, "one" and/or "an" should be interpreted as "at least one" or "one or more"; the same is true for the use of an attributive to introduce a claim statement. In addition, even if a specific number of the introduced claim statements is explicitly stated, a person of ordinary skill in the art will recognize that such a statement should be interpreted as referring to at least the number stated; for example, a direct statement of "two statements" without other modifiers refers to at least two statements, or two or more statements. In addition, in those cases where a convention similar to "at least one of A, B, and C, etc." is used, generally speaking, such a structure is performed in the conventional sense understood by a person of ordinary skill in the art. For example, "a system having at least one of A, B, and C" includes but is not limited to a system having A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc. One of ordinary skill in the art will further understand that almost any non-conjunction and/or phrase presenting two or more alternative terms, whether in the description, patent scope, or drawings, should be understood to consider the possibility of including one of the terms, any one of the terms, or both of the terms. For example, the phrase "A or B" will be understood to include the possibility of "A" or "B" or "A and B".

As can be seen from the above, various embodiments of the present application have been described here for the purpose of illustration, and various modifications may be made without departing from the scope and spirit of the present application. Therefore, the various embodiments disclosed here are not meant to be restrictive, and the true scope and spirit are represented by the following patent scope.

100, 400: current block 500: data path 505: corresponding input component samples 510: matrix preparation module 515: autocorrelation matrix and cross-correlation vector 530: matrix equation solver module 535: optimization coefficients 540: coefficient constraint module 545: constraint coefficients 550: component prediction model 560: reference component samples 565: prediction component samples 600: video encoder 605: video source 608: subtractor 609: prediction residual 610: transform module 611: quantization module 612, 912: quantization coefficients 613, 913: prediction pixel data 614, 911: Inverse quantization module 615, 910: Inverse transform module 616, 916: Transformation coefficients 617: Reconstructed image data 619: Reconstructed residual 620: Intra-image estimation module 625, 925: Intra-frame prediction module 630, 930: Motion compensation module 635: Motion estimation module 640: Inter-frame prediction module 645, 945: Intra-loop filter 650, 950: Reconstructed image buffer 665, 965: MV buffer 675, 975: MV prediction module 690: Entropy encoder 695, 995: Bitstream 705, 1005: Model builder 710, 1010: component prediction model 715, 1015: initial predictor 720, 1020: initial predictor generation module 725, 1025: fine predictor 730, 1030: regression data selection module 735: constraint 800, 810, 820, 830, 840, 850: step 900: video decoder 940: inter-frame prediction module 990: parser (entropy decoder) 917: decoded pixel data 919: reconstructed residual signal 955: display device 1100, 1110, 1120, 1130, 1140, 1150: step 1200: electronic system 1205: bus 1210: processing unit 1215: graphics processing unit (GPU) 1220: system memory 1225: network 1230: read-only memory 1235: permanent storage device 1240: input device 1245: output device

The following figures are included to provide a further understanding of the present application and are incorporated into and constitute a part of the present application. The figures illustrate embodiments of the present application and, together with the description, are used to explain the principles of the present application. It is worth noting that the figures are not necessarily drawn to scale, and some components may be shown as being out of proportion to the size of the actual implementation in order to clearly illustrate the concepts of the present application. FIG. 1 conceptually illustrates chrominance and luminance samples used to derive linear model parameters. FIG. 2 shows an example of classifying adjacent samples into two groups. FIG. 3 conceptually illustrates the spatial components of a convolution filter. FIG. 4 shows a reference region for filter coefficients used to derive a convolution model of the current block. FIG. 5 conceptually illustrates a data path of a video codec that derives and constrains the coefficients of the model. FIG. 6 illustrates an exemplary video encoder that can encode a block of pixels using a component prediction model. FIG. 7 illustrates a portion of a video encoder that derives and uses a component prediction model by constraining coefficients of the model. FIG. 8 conceptually illustrates a process for constraining coefficients of a component prediction model. FIG. 9 illustrates an example of a video decoder that can decode a block of pixels using a component prediction model. FIG. 10 illustrates a portion of a video decoder that derives and uses a component prediction model by constraining coefficients of the model. FIG. 11 conceptually illustrates a process for constraining coefficients of a component prediction model. FIG. 12 conceptually illustrates an electronic system for implementing some embodiments of the present application.

800, 810, 820, 830, 840, 850: Steps

Claims (15)

A video encoding and decoding method, the improvement of which comprises: receiving data of a pixel block to be encoded or decoded as a current block of a current picture of a video; deriving a coefficient set based on corresponding input and output component samples; constraining the derived coefficient set according to a set of constraint conditions; applying the constrained coefficient set as a component prediction model to generate a predictor for the current block; and encoding or decoding the current block by using the generated predictor. A method as described in claim 1, wherein the component prediction model is a cross-component model that generates predicted chrominance samples based on reconstructed luminance samples of the current block. The method of claim 1, wherein the corresponding input and output component samples are corresponding luminance and chrominance samples of a template area adjacent to the current block. A method as described in claim 1, wherein the component prediction model is a convolution model; the coefficient set is derived by solving a matrix equation between the corresponding input and output component samples. The method as described in claim 1, wherein the step of constraining the derived set of coefficients includes: clipping the coefficients at a clipping threshold. The method as described in claim 1, wherein the step of constraining the derived coefficient set includes: clipping different coefficients at different clipping thresholds. A method as described in claim 1, wherein the derived coefficient set and the constrained coefficient set are represented in floating point in a video codec. A method as described in claim 1, wherein the step of constraining the derived set of coefficients comprises: limiting the coefficients to a predefined range. A method as described in claim 8, wherein the derived coefficient set and the constrained coefficient set are represented in a video codec by a fixed point representation whose fractional portion comprises N bits and the size of the predefined range is magnified based on 1<<N. A method as described in claim 8, wherein when the derived coefficients are outside the predefined range, the set of coefficients is set equal to the identification filter. A method as described in claim 8, wherein when the derived coefficient is outside the predetermined range, a clipping operation is applied to the coefficient outside the range. A method as described in claim 11, wherein when the derived coefficients are outside the predetermined range, the derived set of coefficients is not used to encode or decode the current block. An electronic device, the improvement of which comprises: A video codec circuit, configured to perform operations including: Receiving data of a block of pixels to be encoded or decoded as a current block of a current picture of a video; Derived a coefficient set based on corresponding input and output component samples; Constrained the derived coefficient set according to a set of constraint conditions; Applying the constrained coefficient set as a component prediction model to generate a predictor for the current block; and Encoding or decoding the current block by using the generated predictor. A video decoding method, the improvement of which comprises: receiving data of a pixel block to be decoded as a current block of a current picture of a video; deriving a coefficient set based on corresponding input and output component samples; constraining the derived coefficient set according to a set of constraint conditions; applying the constrained coefficient set as a component prediction model to generate a predictor for the current block; and reconstructing the current block by using the generated predictor. A video coding method, the improvement of which comprises: receiving data of a block of pixels to be coded as a current block of a current picture of a video; deriving a set of coefficients based on corresponding input and output component samples; constraining the derived set of coefficients according to a set of constraint conditions; applying the constrained set of coefficients as a component prediction model to generate a predictor for the current block; and encoding the current block by using the generated predictor.