CN111479110B - Fast Affine Motion Estimation Method for H.266/VVC - Google Patents

Abstract

The invention provides a fast affine motion estimation method aiming at H.266/VVC, which comprises the following steps: calculating the texture complexity of the current CU by using the standard deviation, and dividing the current CU into a static area or a non-static area according to the texture complexity; for a CU in a static area, skipping affine motion estimation, directly predicting the current CU by using the motion estimation, and selecting an optimal prediction direction mode by a rate distortion optimization method; and for the CU in the non-static area, classifying the current CU by using a trained random forest classifier RFC model, and outputting the best prediction direction mode. For the CU in the static area, affine motion estimation is skipped, and the calculation complexity is reduced; for the CU in the non-static area, the prediction of the prediction direction mode is directly carried out through the model trained in advance, the calculation of affine motion estimation is avoided, and therefore the complexity of an affine motion estimation module is reduced.

Description

Fast Affine Motion Estimation Method for H.266/VVC

technical field

The present invention relates to the technical field of image processing, in particular to a fast affine motion estimation method for H.266/VVC.

Background technique

In today's information age, the demand for video services such as 3D images, ultra-high-definition video, and virtual reality is increasing, and the coding and transmission of high-definition video has increasingly become a hot research issue. With the development and improvement of the H.266/VVC standard, the improvement of video processing efficiency has also driven the development of the video industry, laying the foundation for the development of a new generation of video coding technology. Highly dense data brings huge challenges to bandwidth and storage. The current mainstream video coding standards cannot meet the current emerging applications. Therefore, a new generation of video coding standards H.266/VVC came into being to meet people's requirements for video clarity. , fluency and real-time requirements. The International Organization for Standardization ISO/IEC MPEG and ITU-T VCEG established the Joint Video Exploration Team (JVET), responsible for the development of the next-generation video coding standard H.266/Versatile Video Coding (VVC). The formulation of H.266/VVC is aimed at 4K and above high-definition video, and the bit depth is mainly 10 bits. This is different from the positioning of H.265/HEVC, which leads to the current maximum block size of the encoder becoming 128. The processed pixels are all 10-bit, even if the input 8-bit sequence will be converted to 10-bit for processing.

H.266/VVC uses a hybrid coding technology framework, and the image division is constantly developing from a single, fixed division to a diverse and flexible division structure, which can more efficiently adapt to the encoding and decoding processing of high-resolution images. In addition, H.266/VVC extends the original H.265/HEVC encoder's inter-intra prediction, prediction signal filtering, transformation, quantization/scaling, entropy coding and other new elements for the new generation of video data, and Considering the characteristics of next-generation video coding standards, a new model prediction mode has been added. Specifically, H.266/VVC follows the motion estimation, motion compensation, and motion vector prediction technologies of H.265/HEVC inter-frame coding in high-efficiency video coding, and introduces some new technologies on this basis. For example, the Merge mode is extended to add prediction motion vector based on history, and new prediction methods such as affine transformation technology, adaptive motion vector precision method, motion prediction compensation with 1/16 sampling precision, etc. are added. The introduction of many advanced coding tools has greatly improved the coding efficiency of the new generation video coding standard H.266/VVC. But it also significantly increases the computational complexity of H.266/VVC inter-frame coding due to rate-distortion cost calculation, thereby significantly reducing the coding speed of the new generation of video.

The main principle of inter-frame prediction is to find the best matching block in the previously encoded image for each pixel block of the current image. This process is called motion estimation ME, where the image used for prediction is called the reference image. The block is the best matching block in the reference image, that is, the reference pixel block, the displacement from the reference block to the current pixel block is called the motion vector MV, and the difference between the current pixel block and the reference block is called the prediction residual. Among them, the motion estimation ME algorithm is the most critical algorithm in the H.266/VVC video encoding process. It occupies more than half of the calculation amount and most of the calculation time of the entire video encoding, and is the dominant factor determining the video compression efficiency. Motion estimation ME has become a research hotspot in video compression technology by effectively removing temporal redundancy between consecutive images. To improve compression efficiency, recent video codecs try to estimate motion of different shapes and sizes. Furthermore, by adding multi-type trees, motion estimation ME can be performed on very thin blocks (e.g., width is one-eighth of height). Therefore, among the various modules of multi-type trees (MTT), motion estimation ME is the tool with the highest coding complexity in VVC. Due to the recursive execution of more advanced inter-frame prediction schemes in the finely divided blocks of MTT, the computational complexity of motion estimation ME is even more than that of HEVC, because ME in Future video coding (Future videocoding, FVC) also tries to imitate New technologies such as shot motion estimation. Affine Motion Estimation AME, characterized by non-translational motions such as rotation and scaling, is effective in rate distortion (RD) performance at the expense of higher encoding complexity. In the whole processing time of motion estimation ME, the computational complexity of affine motion estimation AME accounts for a large part, so it is very important to reduce its complexity. Therefore, to reduce the complexity of the VTM encoder, it is necessary to speed up the AME module.

In fact, a lot of research has been done in many literatures on the high complexity of H.265/HEVC inter-frame prediction. J. Xiong et al. proposed a fast CU selection algorithm based on the vertebral body motion divergence, which can skip individual inter-frame CUs in H.265/HEVC in advance. L.Shen et al. proposed an adaptive inter-mode decision algorithm for H.265/HEVC. This algorithm jointly utilizes inter-layer and spatio-temporal correlation, and proposes an early skip mode decision based on statistical analysis. Based on the prediction size Correlation mode decision and mode decision based on rate-distortion (RD) cost correlation are three methods. H.Lee et al. proposed an early skip mode decision-making method by using its distortion characteristics after calculating the RD cost of the 2N×2N Merge mode. Aiming at the high correlation between texture video and depth map content, Q. Zhang et al. proposed an early decision coding unit depth level and adaptive mode decision method to reduce the computational complexity of video coding. Q.Hu et al proposed a fast inter-frame mode decision algorithm based on the Neyman-Pearson rule, which includes early SKIP mode decision and fast CU size decision to reduce the complexity of H.265/HEVC. Z. Pan et al. proposed a fast reference frame selection algorithm based on content similarity to reduce the computational complexity of inter-frame prediction based on multiple reference frames. Z.Pan et al. proposed a fast motion estimation ME method based on the best motion vector selection correlation between different size prediction modes to reduce the coding complexity of the H.265/HEVC encoder. J. Zhang et al. proposed a two-stage fast inter-frame CU decision method based on Bayesian method and conditional random field to reduce the coding complexity of HEVC encoder. T.S.Kim et al. proposed a HEVC-based fast motion estimation algorithm that supports a highly flexible block partition structure. The algorithm greatly reduces its computational complexity by searching a narrow region around multiple accurate motion vector predictions. C. Ma et al. proposed a neural network-based arithmetic coding method to encode inter-frame prediction information in HEVC. L.Shen et al proposed a fast mode decision algorithm to reduce the computational complexity of the encoder. The proposed method utilizes three optimized encoders with tuned parameters, namely conditional probability of SKIP/Merge modes, motion characteristics and mode complexity. D.Wang et al. proposed a fast depth-level and inter-mode prediction algorithm. This algorithm uses inter-layer correlation, spatial correlation and their degree of correlation to speed up HEVC inter-coding. The above fast inter-frame methods can effectively reduce the computational complexity of H.265/HEVC while ensuring the coding performance. However, these methods are not designed for H.266/VVC encoders, and H.266/VVC encoders use new inter-frame prediction techniques, such as more advanced affine motion compensation prediction, extended Merge mode, adaptive Motion vector accuracy, triangulation mode and other technologies. Based on this, the spatial and inter-layer correlation between H.266/VVC and H.265/HEVC-based inter-frame prediction must be quite different, so it is necessary to re-study the low-complexity inter-frame coding based on H.266/VVC method.

Aiming at the high complexity of H.266/VVC inter-frame coding, very few literatures have explored it. S.Park et al. proposed a method to effectively limit the search range of reference frames for normal motion estimation and affine motion estimation, which mainly utilizes the dependence within the H.266/VVC prediction structure. This method minimizes the maximum value of the reference frame search range of the CU based on the prediction information of the parent node to reduce the coding complexity. Z.Wang et al. proposed an early termination scheme based on a confidence interval-based quadtree plus binary tree (QTBT) partition structure, and established a rate distortion (RD) model based on the motion divergence field. To estimate the rate-distortion RD cost of each partition mode; and based on this model, the block division of H.266/VVC is terminated early to eliminate unnecessary partition iterations, making the relationship between H.266/VVC coding performance and coding complexity strike a good balance. Z.Wang et al. proposed a fast QTBT partition decision algorithm for convolutional neural networks (CNN) for H.266/VVC inter-frame coding. The algorithm analyzes QTBT in a statistical way to design Architecture of Convolutional Neural Network (CNN) and exploiting temporal correlation to control misprediction risk to improve the robustness of Convolutional Neural Network (CNN) schemes. D.García-Lucas et al proposed a pre-analysis algorithm for extracting frame motion information, which is used in the motion estimation module to accelerate the H.266/VVC encoder. S. Park et al proposed a fast H.266/VVC interframe coding method to effectively reduce the coding complexity of affine motion estimation in VTM when using multi-type tree MTT. The method consists of two processes: an early termination scheme and reducing the number of reference frames for affine motion estimation. H.Gao et al. proposed a low-complexity decoder-side motion vector refinement scheme, by searching the block with the smallest matching cost in the previously decoded reference picture, optimizing the initial motion vector MV from the Merge mode, and adding In the decoder-side motion vector refinement method based on bilateral matching. N.Tang et al. proposed a fast block division algorithm for H.266/VVC inter-frame coding, using three-frame difference to judge whether the current block is a static object; when the current block is static, no further splitting is required, thereby terminating early Partitioning for faster inter-coding. However, little work has been done to alleviate the AME complexity of affine motion estimation in VVC. For VTM, there is a lot of room to further reduce the ME complexity of motion estimation in multi-type tree MTT structure, especially in affine motion estimation AME.

Contents of the invention

Aiming at the deficiencies in the above-mentioned background technology, the present invention proposes a fast affine motion estimation method for H.266/VVC, which solves the technical problem of high AME encoding complexity of affine motion estimation in VTM.

Technical scheme of the present invention is realized like this:

A fast affine motion estimation method for H.266/VVC, the steps are as follows:

S1. Calculate the texture complexity SD of the current CU by using the standard deviation, and divide the current CU into a static area or a non-static area according to the texture complexity SD;

S2. For the CU in the static area, skip the affine motion estimation AME, directly use the motion estimation CME to predict the current CU, and select the best prediction direction mode through the method of rate-distortion optimization;

S3. For the CU in the non-static area, use the trained random forest classifier RFC model to classify the current CU, and output the best prediction direction mode.

The method for calculating the texture complexity SD of the current CU by using the standard deviation is:

Among them, W represents the width of the CU, H represents the height of the CU, and P(a, b) represents the pixel value at the position (a, b) in the CU.

The method of using motion estimation CME to predict the current CU and selecting the best prediction direction mode through the method of rate-distortion optimization is as follows:

S21. The current CU first undergoes unidirectional prediction Uni-L0, then unidirectional prediction Uni-L1, and finally bidirectional prediction Bi;

S22. Utilization rate-distortion optimization calculates the rate-distortion cost of the current CU in step S21 respectively through unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi;

S23. Taking the prediction mode with the smallest rate-distortion cost as the best prediction direction mode.

The calculation methods of the rate-distortion cost of the unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi are:

表示所有可用参考列表集合，

表示参考列表集，L0和L1表示两个参考帧列表，φ(j)表示参考列表中的参考帧，J(·)为率失真代价函数，且

D(·)表示CU编码的失真程度，λ表示拉格朗日乘子，R(·)表示CU编码消耗的比特数。in,

represents the set of all available reference lists,

Represents the reference list set, L0 and L1 represent two reference frame lists, φ(j) represents the reference frame in the reference list, J( ) is the rate-distortion cost function, and

D(·) represents the degree of distortion of CU coding, λ represents the Lagrangian multiplier, and R(·) represents the number of bits consumed by CU coding.

The training method of the random forest classifier RFC model in the described step S3 is:

S31. Select Traffic, Kimono, BQSquare, RaceHorseC, and FourPeople video sequences at different resolutions from the general test sequence, respectively encode the first M frames on the VTM, and record the shape of the CU in the VTM, the texture complexity of the CU, and the CU The three prediction direction modes of are used as a data set, and the data set includes a sample set S and a test set T, where the three prediction direction modes include unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi;

将生成的每个训练集

作为根节点，生成对应的决策树{T₁,T₂,...,T_K}，其中，i＝1,2,…,K表示第i个训练样本，K表示训练样本集的大小；S32. Using the Bootstrap method to resample the sample set S to generate K training sample sets

Each training set that will be generated

As the root node, generate a corresponding decision tree {T ₁ , T ₂ ,...,T _K }, where i=1, 2,..., K represents the i-th training sample, and K represents the size of the training sample set;

S33, start training from the root node, randomly select m feature attributes on each intermediate node of the decision tree, calculate the Gini index coefficient of each feature attribute, and select the feature attribute with the smallest Gini index coefficient as the optimal split of the current node attribute, with the minimum Gini index coefficient as the splitting threshold, divide m feature attributes into left subtree and right subtree;

S34, repeating step S33, training K' times, until the K' decision tree training is completed, and each decision tree is fully grown without pruning;

S35. The multiple decision trees generated are the random forest classifier RFC model, and the random forest classifier RFC model is used to discriminate and classify the test set T, and the classification result adopts the voting method, and the category with the most output of K' decision trees is used as The category of the test set T to obtain the best prediction direction mode of the current CU.

The method for obtaining the data set in the step S31 is:

S31.1. Using motion estimation CME to predict the video sequence;

S31.2. Use the 4-parameter affine motion model to perform affine prediction on the video sequence predicted in step S31.1, wherein the affine prediction includes unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi ;

S31.3, using the 6-parameter affine motion model to perform radial prediction on the video sequence after the affine prediction in step S31.2;

S31.4. Calculate the rate-distortion cost after the affine prediction in steps S31.2 and S31.3 respectively, and set the prediction mode corresponding to the smallest rate-distortion cost as the prediction direction mode of the video sequence.

The feature attributes include two-dimensional Haar wavelet transform horizontal coefficients, two-dimensional Haar wavelet transform vertical coefficients, two-dimensional Haar wavelet transform angle coefficients, second-order moments of angles, contrast, entropy, inverse moments, minimum difference sums and gradients .

In the 4-parameter affine motion model, the motion vector of the sample position (x, y) in the CU is:

Among them, (mv _0x , mv _0y ) is the motion vector of the control point in the upper left corner, (mv _1x , mv _1y ) is the motion vector of the control point in the upper right corner, and W represents the width of the CU;

In the 6-parameter affine motion model, the motion vector of the sample position (x, y) in the CU is:

Wherein, (mv _2x , mv _2y ) is the motion vector control point in the lower left corner, and H represents the height of the CU.

Beneficial effects that can be produced by this technical solution: the present invention first uses the standard deviation SD to divide the CU into a static area and a non-static area. If the CU belongs to the static area, the probability of selecting SKIP mode for inter-frame prediction is relatively high, and SKIP tends to be selected Affine prediction is not required for the static area where inter-frame prediction is performed. Therefore, the affine motion estimation AME module can be terminated early in the static area, and the best direction mode of the current CU is the best direction mode of the motion estimation CME; if the CU If it belongs to a non-static area, the inter-frame prediction mode of the CU is judged according to the random forest classification model, and finally the optimal prediction direction mode is obtained in advance; therefore, the present invention reduces the computational complexity and saves encoding time, thereby realizing H.266/ Fast encoding for VVC.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a flowchart of the present invention;

Fig. 2 is the distribution diagram of the prediction direction pattern complexity of the present invention;

Fig. 3 is 4-parameter affine model of the present invention;

Fig. 4 is 6-parameter affine model of the present invention;

FIG. 5 is an overall process diagram of the motion estimation ME of the present invention;

Fig. 6 is a comparison result graph of the overall running time between the method of the present invention and the FAME method.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

As shown in Figure 1, the embodiment of the present invention provides a fast affine motion estimation method for H.266/VVC, and the specific steps are as follows:

S1. During the image coding process, the image content of a single region is often coded using a relatively large CU. In contrast, regions with rich details are usually encoded with smaller CUs. It can be seen that whether the CU uses the SKIP mode for inter-frame prediction is determined by the degree of texture complexity of the coding block. In the image coding process, areas with single image content tend to use SKIP mode for inter-frame prediction for encoding, while areas with rich details are less likely to use SKIP mode for inter-frame prediction. The variance of a CU represents the degree of energy dispersion between two pixels of the current block, so the texture complexity of a block can be roughly measured by its standard deviation SD. Therefore, the standard deviation is used to calculate the texture complexity SD of the current CU, and According to the texture complexity SD, the current CU is divided into a static area or a non-static area; the formula for the standard deviation is:

Among them, W represents the width of the CU, H represents the height of the CU, and P(a, b) represents the pixel value at the position (a, b) in the CU. Since the texture complexity of adjacent blocks is correlated with the CU, the threshold for classification is derived from the texture complexity of adjacent blocks. According to a large amount of experimental data, it is reasonable to take the minimum value of the standard deviation SD of adjacent blocks of the CU as Th _static . CUs can be classified by thresholds. If the current standard deviation SD is smaller than the threshold Th _static , it indicates that the current CU is a static area. On the contrary, if the value of the standard deviation SD is greater than Th _static , the current CU belongs to the non-static area.

S2. Existing video coding standards (such as H.265/HEVC) use motion vector MV covering translational motion for motion estimation CME, however, affine motion estimation AME can predict not only translational motion, but also linear transformation motion, such as scaling and rotation. If the camera is scaled or rotated to capture the video, affine motion estimation (AME) predicts motion more accurately than motion estimation (CME). In H.266/VVC, the affine motion estimation AME is the same as the motion estimation CME, starting from the unidirectional prediction Uni-prediction L0, then the unidirectional prediction Uni-prediction L1, and finally the bidirectional prediction Bi-prediction. After calculating the three predicted direction modes, a rate distortion optimization (RDO) method is used to select the best predicted direction mode. Fig. 2 shows the distribution of the complexity of AME inter-prediction modes, and Uni-prediction L0 requires more prediction time than Uni-prediction L1. If the reference frames of Uni-prediction L0 and Uni-prediction L1 are different, the coding complexity of Uni-prediction is the same. Otherwise, if the reference frame between Uni-prediction L0 and Uni-prediction L1 is the same, the motion vector MV of Uni-prediction L1 is copied from the motion vector of Uni-prediction L0, To avoid redundant affine motion estimation AME processing. Therefore, Uni-prediction L0 prediction consumes more prediction time than Uni-prediction L1. Although the greatest coding complexity is required in unidirectional prediction, the bidirectional prediction mode has a higher probability as the best inter prediction mode. In the AME module of Affine Motion Estimation, calculating the rate-distortion RD cost is an important reason for the high complexity.

其中，D(·)表示CU编码的失真程度，λ表示拉格朗日乘子，R(·)表示CU编码消耗的比特数。因为用单向预测Uni-L0和单向预测Uni-L1表示的两个参考帧列表用于运动预测，所以应该用两个列表测试用于单向预测的运动估计ME过程，从而在两个列表中生成所有可用的帧。In order to obtain the best motion vector MV and the best reference frame, the encoder searches multiple available reference frames, calculates the rate-distortion RD cost J( ) using the Lagrange multiplier method and compares the cost of the prediction results, Lagrangian The daily multiplier method calculates the rate-distortion RD cost function J(·) as:

Among them, D(·) represents the degree of distortion of CU coding, λ represents the Lagrangian multiplier, and R(·) represents the number of bits consumed by CU coding. Since two lists of reference frames denoted by Uni-L0 and Uni-L1 are used for motion prediction, the motion estimation ME process for Uni-prediction should be tested with Generate all available frames in .

For the CU in the static area, skip the affine motion estimation AME, directly use the motion estimation CME to predict the current CU, and select the best prediction direction mode through the method of rate-distortion optimization; the specific method is:

S21. The current CU first undergoes unidirectional prediction Uni-L0, then undergoes unidirectional prediction Uni-L1, and finally undergoes bidirectional prediction Bi.

S22. Utilization rate-distortion optimization calculates the rate-distortion cost after unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi respectively;

The rate-distortion costs of the unidirectional prediction Uni-L0, unidirectional prediction Uni-L1, and bidirectional prediction Bi are respectively:

表示所有可用参考列表集合，

表示参考列表集，L0和L1表示两个参考帧列表，φ(j)表示参考列表中的参考帧，J(·)为率失真代价函数。in,

represents the set of all available reference lists,

Represents the reference list set, L0 and L1 represent two reference frame lists, φ(j) represents the reference frame in the reference list, and J(·) is the rate-distortion cost function.

S23. Taking the prediction mode with the smallest rate-distortion cost as the best prediction direction mode.

S3. For the CU in the non-static area, the condition of skipping the AME process of affine motion estimation is not met, and the current CU is classified by using the trained random forest classifier RFC model, and the best prediction direction mode is output to further reduce Computational complexity. The random forest algorithm generates K self-service sample sets based on Bootstrap resampling, and the data of each sample set grows into a decision tree; at the node of each tree, based on the random subspace method RSM, randomly select from M' feature vectors Extract m(m<<M') features. According to a certain node splitting algorithm, the optimal attribute is selected from m feature attributes for branch growth; finally, K'decision trees are combined for majority voting. After the random forest classifier is generated, the random forest classifier model is tested. Each tree in the forest will independently judge the classification results, and the final decision will take the classification category with the most identical judgments. The formula is expressed as follows,

Among them, H(t) represents the combined classification model, h _i (t) is a single classification tree model, t represents the characteristic attribute of the decision tree, Y represents the output variable, and I( ) represents the set indicative function (that is, when the set When there is a classification result, the function value is 1, otherwise it is 0).

When traversing a CU, record the characteristics of the CU and the prediction direction mode of the CU, without interfering with the normal encoding process. Multiple training sets are generated by resampling through the Bagging integration method, random and equal samples are drawn from the original training sample set, and K new training sample sets are generated by repeated extraction with replacement, and finally K new training sample sets are obtained. After the sample is extracted, it enters the training module of the random forest classifier model. Table 1 shows the relevant training parameter settings established by the random forest classifier RFC model.

Table 1 Training parameter configuration

According to the parameters in Table 1, the training method of the random forest classifier RFC model is:

S31. One of the keys to training a classifier is the selection of sample sets. Select Traffic, Kimono, BQSquare, RaceHorseC, and FourPeople video sequences with different resolutions that can cover rich texture complexity from the general test sequence, and respectively on the VTM M=50 frames before coding, record the shape of the CU in the VTM, the texture complexity of the CU and the three prediction direction modes of the CU as a data set, the data set includes a sample set S=20 and a test set T=30, among which, three The prediction direction modes include unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi;

In VTM, blocks with affine motion are also predicted in three ways: unidirectional prediction Uni-L0, unidirectional prediction Uni-L1, and bidirectional prediction Bi. Meanwhile, affine prediction also includes 4-parameter and 6-parameter affine models. Affine Motion Estimation The unidirectional prediction or bidirectional prediction of the AME module requires related reference frames, which increases the coding complexity of VTM. The affine motion estimation AME process requires twice as many reference frames as the motion estimation CME process when only the number of reference frames required by each affine motion estimation AME module is calculated. The whole ME process of motion estimation is shown in Fig. 5. As can be seen from Figure 5, the method for obtaining the data set in step S31 is:

S31.1. Using motion estimation CME to predict the video sequence, the prediction method is the same as step S21;

S31.2. Use the 4-parameter affine motion model to perform affine prediction on the video sequence predicted in step S31.1, wherein the affine prediction includes unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi ;

As shown in Figure 3, the motion vector of the sample position (x, y) in the CU of the 4-parameter affine motion model is:

Among them, (mv _0x , mv _0y ) is the motion vector of the control point in the upper left corner, (mv _1x , mv _1y ) is the motion vector of the control point in the upper right corner, and W represents the width of the CU;

S31.3, using the 6-parameter affine motion model to perform radial prediction on the video sequence after the affine prediction in step S31.2;

As shown in Fig. 4, the motion vector of the sample position (x, y) in the block of the 6-parameter affine motion model is:

Wherein, (mv _2x , mv _2y ) is the motion vector control point in the lower left corner, and H represents the height of the CU.

S31.4. Calculate the rate-distortion cost after the affine prediction in steps S31.2 and S31.3 respectively, and set the prediction mode corresponding to the smallest rate-distortion cost as the prediction direction mode of the video sequence.

将生成的每个训练集

作为根节点，生成对应的决策树{T₁,T₂,...,T_K}，其中，i＝1,2,…,K表示第i个训练样本，K表示训练样本集的大小；S32. Using the Bootstrap method to resample the sample set S to generate K training sample sets

Each training set that will be generated

As the root node, generate a corresponding decision tree {T ₁ , T ₂ ,...,T _K }, where i=1, 2,..., K represents the i-th training sample, and K represents the size of the training sample set;

S33, start training from the root node, randomly select m feature attributes on each intermediate node of the decision tree, calculate the Gini index coefficient of each feature attribute, and select the feature attribute with the smallest Gini index coefficient as the optimal split of the current node attribute, with the minimum Gini index coefficient as the splitting threshold, divide m feature attributes into left subtree and right subtree;

The effectiveness of machine learning is highly related to the diversity and relevance of the training dataset. Although the random forest classifier RFC can handle ultra-high-dimensional feature data, selecting truly relevant feature vectors can better generalize the classification model. Since the prediction direction mode of the CU is related to the texture, texture direction and motion state of the image, these are used as the classification basis, that is, as the feature vector of the random forest classifier model. The feature attributes selected by the present invention include two-dimensional Haar wavelet transform horizontal coefficient (2D Haar wavelet transform horizontal coefficient, HL), two-dimensional Haar wavelet transform vertical coefficient (2D Haar wavelet transform vertical coefficient, LH), two-dimensional Haar wavelet transform Transform angle coefficient (2D Haar wavelet transform angle coefficient, HH), angular second moment (angular second moment, ASM), contrast (contrast, CON), entropy (entropy, ENT), inverse difference moment (inverse difference moment, IDM), The minimum difference and (Sum of Absolute Difference, SAD) and gradient (gradient) are used as the characteristic attributes of the random forest classifier model, and the calculation of the characteristic attributes is as follows:

The horizontal coefficient HL of the two-dimensional Haar wavelet transform of the image represents the texture in the horizontal direction of the image. The larger the value, the richer the texture in the horizontal direction, and the smaller the value, the flatter the texture in the horizontal direction; the vertical coefficient of the two-dimensional Haar wavelet transform of the image LH represents the texture in the vertical direction of the image, the larger the value, the richer the texture in the vertical direction, the smaller the value, the flatter the texture in the vertical direction; the two-dimensional Haar wavelet transform angle coefficient HH of the image represents the texture in the vertical direction of the image, the smaller the value A larger value indicates that the texture in the 45° direction is richer, and a smaller value indicates that the texture in the 45° direction is flatter. The two-dimensional Haar wavelet transform horizontal coefficient HL, the two-dimensional Haar wavelet transform vertical coefficient LH and the two-dimensional Haar wavelet transform angle coefficient HH are expressed as:

Among them, W represents the width of the CU, H represents the height of the CU, and P(a, b) represents the pixel value at the position (a, b).

Angular second moment ASM reflects the uniformity of gray distribution and texture thickness. The larger the value, the more uniform the texture distribution of the image; the contrast CON reflects the texture depth of the image. The larger the value, the greater the texture depth; the entropy ENT represents the amount of information in the image , the larger the value, the greater the information content of the image; the inverse difference moment IDM reflects the size of the local texture change of the image, the texture of the image is relatively uniform between different regions, and the change is slow, the second-order moment of angle ASM, contrast CON, entropy ENT and inverse difference moment IDEs are represented as:

In the motion estimation algorithm based on block matching, there are many criteria for judging the best matching block. We use the minimum difference and SAD. The smaller the SAD, the closer the reference block is to the current prediction block. The minimum difference and SAD are expressed as:

Among them, P _k (a, b) represents the value of the current pixel, (a, b) represents the coordinates of the current pixel, P _k-1 (a+i', b+j') is the reference pixel value, (a+i ',b+j') represent the coordinates of the reference pixel.

The gradient represents the texture direction of the CU, using the horizontal and vertical gradients of luma samples as feature attributes. The gradients in the horizontal and vertical directions are expressed as:

G _x (a,b)=P(a+1,b)-P(a,b)+P(a+1,b+1)-P(a,b+1),

G _y (a,b)=P(a,b)-P(a,b+1)+P(a+1,b)-P(a+1,b+1),

Among them, G _x (a, b) and G _y (a, b) represent the gradient components of the current pixel in the horizontal and vertical directions, respectively. (a,b) represents the coordinates of the pixel, and P(a,b) represents the pixel value.

S34, step S33 is repeated, training K'=25 times, until the K' decision tree training is completed, and each decision tree is fully grown without pruning;

S35. The multiple decision trees generated are the random forest classifier RFC model, and the random forest classifier RFC model is used to discriminate and classify the test set T, and the classification result adopts the voting method, and the category with the most output of K' decision trees is used as The category of the test set T, the best prediction direction mode of the current CU is obtained, and the computational complexity of the affine motion estimation AME module is reduced.

In order to evaluate the method of the present invention, simulation tests are carried out on the latest H.266/VVC encoder (VTM 7.0). The test video sequences were encoded with default parameters in the "Random Access" configuration. BDBR reflects the compression performance of the present invention, and the reduction in time reflects the reduction in complexity. Table 2 shows the encoding characteristics of the present invention, the total encoding time of the present invention is reduced to 87% on average, and the AME time of affine motion estimation is reduced to 56% on average. Therefore, the present invention can effectively save encoding time, and the loss of RD performance can be neglected.

Encoding characteristic of the present invention of table 2

It can be seen from Table 2 that the present invention has RD performance and saved encoding running time compared with VTM. For different test videos, the experimental results may fluctuate, but the method proposed by the present invention is effective. Compared with VTM, the invention can effectively reduce the complexity of AME module of affine motion estimation, and has good RD performance.

Affine motion estimation AME module time is measured according to different quantization parameters (Quantization parameter, QP). When the quantization parameter QP is 22, it can be seen from Fig. 6 that the total time of AME module for all video sequences is about 36 hours. However, in the method of the present invention, the time of the affine motion estimation AME module is reduced by about 9 hours. It can be seen that this trend is similar under other quantization parameters QPs. Therefore, it is more intuitively observed from Fig. 6 that the proposed method reduces the encoding time of the AME module for affine motion estimation, thereby reducing the computational complexity.

The technical solution of the present invention has been described in detail above in conjunction with the accompanying drawings. The technical solution of the present invention proposes a fast affine motion estimation method for H.266/VVC, which effectively reduces the AME encoding of affine motion estimation in VTM. the complexity. First, the standard deviation SD is used to divide the CU into a static area and a non-static area. If the CU belongs to a static area, the probability of selecting SKIP mode for inter-frame prediction is higher, and the static area that tends to select SKIP mode for inter-frame prediction does not need to be performed. Affine prediction, therefore, the affine motion estimation AME module can be terminated early in the static region, and the best direction mode of the current CU is the best direction mode of the motion estimation CME. If the CU belongs to a non-static area, the inter-frame prediction mode of the CU is judged according to the random forest classification model, and finally the optimal prediction direction mode is obtained in advance. Therefore, the present invention reduces computational complexity and saves coding time, thereby realizing fast coding of H.266/VVC.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (8)

1. A fast affine motion estimation method for H.266/VVC is characterized by comprising the following steps:

s1, calculating texture complexity SD of a current CU by using a standard deviation, and dividing the current CU into a static area or a non-static area according to the texture complexity SD;

s2, for the CU in the static area, skipping affine motion estimation AME, directly predicting the current CU by using motion estimation CME, and selecting the optimal prediction direction mode by a rate distortion optimization method;

and S3, classifying the current CU by using the trained random forest classifier RFC model for the CU in the non-static area, and outputting the best prediction direction mode.

2. The method of claim 1, wherein the calculating the texture complexity SD of the current CU using the standard deviation is:

where W represents the width of the CU, H represents the height of the CU, and P (a, b) represents the pixel value at position (a, b) in the CU.

3. The fast affine motion estimation method for h.266/VVC as claimed in claim 1, wherein the method of predicting the current CU by using motion estimation CME and selecting the best prediction direction mode by rate distortion optimization is:

s21, firstly performing unidirectional prediction on a current CU through a Uni-L0, then performing unidirectional prediction on a Uni-L1, and finally performing bidirectional prediction on Bi;

s22, respectively calculating the rate distortion cost of the current CU respectively subjected to unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi in the step S21 by using rate distortion optimization;

and S23, taking the prediction mode with the minimum rate distortion cost as the optimal prediction direction mode.

4. The method of claim 3, wherein the rate-distortion cost of Uni-directional prediction Uni-L0 and Uni-directional prediction Uni-L1 are calculated by the following methods:

the method for calculating the rate distortion cost of the Bi-directional prediction Bi comprises the following steps:

where Γ (Φ) represents a set of all available reference lists, Φ represents a set of reference lists, L0 and L1 represent two reference frame lists, Φ (J) represents a reference frame in a reference list, J (-) is a rate distortion cost function, and J (Φ (J)) = D (Φ (J)) + λ · R (Φ (J)), D (-) represents a distortion level of CU coding, λ represents a lagrange multiplier, and R (-) represents a number of bits consumed by CU coding.

5. The fast affine motion estimation method for h.266/VVC according to claim 1, wherein the training method of the RFC model of the random forest classifier in the step S3 is as follows:

s31, selecting Traffic, kimono, BQSquare, raceHorseC and FourPeople video sequences under different resolutions from the universal test sequence, respectively coding a previous M frame on the VTM, and simultaneously recording the shape of a CU, the texture complexity of the CU and three prediction direction modes of the CU in the VTM as a data set, wherein the data set comprises a sample set S and a test set T, and the three prediction direction modes comprise a unidirectional prediction Uni-L0, a unidirectional prediction Uni-L1 and a bidirectional prediction Bi;

s32, resampling the sample set S by using a Bootstrap method, and generating K training sample sets

Each training set to be generated

As root node, generate the corresponding decision tree { T } ₁ ,T ₂ ,...,T _K Where i =1,2, \8230, K denotes the ith training sample, K denotes the size of the training sample set;

s33, training is started from a root node, m characteristic attributes are randomly selected from each intermediate node of the decision tree, a Gini index coefficient of each characteristic attribute is calculated, the characteristic attribute with the minimum Gini index coefficient is selected as the optimal splitting attribute of the current node, the minimum Gini index coefficient is used as a splitting threshold value, and the m characteristic attributes are divided into a left sub-tree and a right sub-tree;

s34, repeating the step S33, training K 'times until the K' decision trees are trained completely, and enabling each decision tree to grow completely without pruning;

s35, the generated decision trees are random forest classifier RFC models, the random forest classifier RFC models are used for distinguishing and classifying the test set T, the classification result adopts a voting mode, the most categories output by the K' decision trees are used as the categories of the test set T, and the best prediction direction mode of the current CU is obtained.

6. The fast affine motion estimation method for h.266/VVC according to claim 5, wherein the method of obtaining data set in step S31 is:

s31.1, predicting a video sequence by utilizing motion estimation CME;

s31.2, carrying out affine prediction on the video sequence predicted in the step S31.1 by utilizing a 4-parameter affine motion model, wherein the affine prediction comprises unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi;

s31.3, carrying out affine prediction on the video sequence subjected to affine prediction in the step S31.2 by using a 6-parameter affine motion model;

and S31.4, respectively calculating rate-distortion costs after affine prediction in the steps S31.2 and S31.3, and taking the prediction mode corresponding to the minimum rate-distortion cost as the prediction direction mode of the video sequence.

7. The method of claim 5 wherein the feature attributes comprise two-dimensional haar wavelet transform horizontal coefficients, two-dimensional haar wavelet transform vertical coefficients, two-dimensional haar wavelet transform angle coefficients, angular second moments, contrast, entropy, inverse moments, minimum difference, and gradient.

8. The fast affine motion estimation method for h.266/VVC as claimed in claim 6, wherein the motion vector of the sample position (x, y) in CU of said 4 parameter affine motion model is:

wherein (mv) _0x ,mv _0y ) Is the motion vector of the upper left corner control point, (mv) _1x ,mv _1y ) Is the motion vector of the top right control point, W represents the CU width;

for the 6-parameter affine motion model, the motion vector of the sample position (x, y) in the CU is:

wherein (mv) _2x ,mv _2y ) Is the motion vector control point in the lower left corner, H denotes the CU's height.

Images