CN116600107B - HEVC-SCC quick coding method and device based on IPMS-CNN and spatial neighboring CU coding modes - Google Patents

Abstract

The application discloses an HEVC-SCC quick coding method and device based on IPMS-CNN and space domain adjacent CU coding modes, which combines a method for predicting a large-size CU mode by a convolutional neural network with a method for predicting a small-size CU mode based on the number of modes adopted by the space adjacent CU, aims at reducing coding time while maintaining coding quality and reducing calculation complexity, and comprises the steps of firstly constructing a database and training a convolutional neural network model selected by an IBC/PLT mode; secondly, the input CTU passes through a mode selection network, and a mode prediction label of the CTU is output; finally, the mode selected by the current CU is predicted by counting the number of modes used by the neighboring 3 CUs. The application can save the encoding time and reduce the calculation complexity of the screen content video.

Description

HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode methods and devices

Technical field

The present invention relates to the field of video coding, and in particular to a HEVC-SCC fast coding method and device based on IPMS-CNN and spatial adjacent CU coding mode.

Background technique

In recent years, with the rapid development of computer vision, multimedia technology and human-computer interaction, screen content video (SCV) applications such as screen sharing, wireless display, and distance education have continued to emerge, which has challenged the traditional processing of natural videos. Video coding methods present huge challenges. Traditional standards for processing natural videos, such as High Effcient Video Coding (HEVC), are specifically developed to compress natural video content captured by cameras. On-screen content video is mainly computer-generated, and it usually features a large area of uniform plane, repeated patterns and characters, a limited variety of colors but high saturation, high image contrast, and sharp edges. If traditional video encoding standards are still used to process screen content videos, the compression effect is often poor. Therefore, in order to take advantage of these special characteristics of screen content videos, the Joint Video Coding Group developed a Screen Content Coding (SCC) standard based on HEVC: HEVC-SCC. The standard adds four new modes: Intra Block Copy (IBC), Palette Mode (PLT), Adaptive Color Transform (ACT), Adaptive Motion Vector Resolution ( Adaptive Motion Vector Resolution (AMVR).

Among these four modes, IBC and PLT are the two main modes that improve compression performance. IBC mode helps to encode repeated patterns within the same frame, while PLT mode aims to encode with a limited number of primary colors. Although the addition of these two tools can significantly improve the coding performance of SCC, its coding complexity also increases significantly.

Contents of the invention

The main purpose of the present invention is to overcome the above-mentioned defects in the existing technology and propose a HEVC-SCC fast encoding method and device based on IPMS-CNN and spatial adjacent CU encoding mode, which can save encoding time while ensuring subjective quality. , reduce the computational complexity of screen content video and accelerate the encoding process of HEVC-SCC.

The present invention adopts the following technical solutions:

On the one hand, a HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode includes:

In the data set production step, video sequence data sets of different resolutions are established and encoded to obtain the real labels of whether each CU of HEVC-SCC uses IBC/PLT mode under different quantization parameters;

The network model construction step is to build a network model IPMS-CNN including an input layer, a feature extraction layer and an output expression layer. Among them, three convolutional layers are built in the feature extraction layer to extract three types of feature maps, plus the feature maps obtained after downsampling. , a total of four feature maps of different sizes are extracted;

The network model training step is to train the constructed network model based on the produced data set to obtain the trained IBC/PLT mode selection convolutional neural network IPMS-CNN model;

In the network model prediction step, the LCU is input into the trained IPMS-CNN to obtain the mode prediction label to predict the mode selection of the CTU;

Current CU mode prediction step, calculates the number of IBC/PLT modes used by three adjacent CUs and the number of Intra modes used by three adjacent CUs , jointly predict the current 8×8CU mode based on the two quantitative relationships;

In the encoding step, the encoder calls the prediction label based on the network model prediction step, and predicts the CU division result together with the current CU mode prediction step.

Preferably, the data set creation steps specifically include:

Self-made video sequence data sets of three different resolutions. The data sets cover image data sets and video data sets, including three types of video sequences: TGM/M, A, and CC;

Then encode through the standard encoding software platform, and set the mode tag of the IBC/PLT mode of each CU under different quantization parameters QP under the full intra-frame configuration.

Preferably, the data set includes: a training set, a verification set and a test set; each of the training set, verification set and test set contains three subsets; the resolution of the first subset is 1024× 576, the second subset has a resolution of 1792×1024, and the third subset has a resolution of 2304×1280.

Preferably, the quantization parameters include four quantization levels, which are 22, 27, 32 and 37 respectively.

Preferably, in the network model building step, three convolutional layers are built in the feature extraction layer to extract three types of feature maps. At the same time, the downsampled feature maps will be directly sent to the connection layer of the network.

Preferably, the output expression layer includes a fully connected layer; a quantization parameter QP is added to the feature vector of the fully connected layer.

Preferably, the loss function of the network model is as follows:

in, Represents the cross entropy of the true value and the predicted value, , , Respectively represent the real mode labels of the first level 64×64, the second level 32×32, and the third level 16×16CU, Real mode tag representing 64×64CTU, , Represents 4 real mode tags of 32×32CTU, Represents 4×4 real mode labels of 16×16CTU; similarly, , , Represents the prediction labels of 64×64 in the first level, 32×32 in the second level, and 16×16 in the third level respectively. Represents the prediction mode label of 64×64CTU, , then represents four prediction mode labels of 32×32CTU, Represents 4 × 4 prediction mode labels of 16 × 16 CTU; both the prediction label and the real label of the network have been binarized, ranging from [0, 1].

Preferably, in the network model prediction step, the network model outputs 21 binary labels indicating whether the 64×64, 32×32, and 16×16 CTUs will be divided and whether the IBC/PLT mode will be selected based on this.

Preferably, the current CU mode prediction step specifically includes:

When the CU size is 8×8, calculate the number of IBC, PLT modes and Intra modes used by three adjacent CUs;

Specifically, when When , the only candidate mode is Intra mode; when and When , the candidate modes are IBC and PLT modes; when and , the candidate modes are Intra, IBC and PLT modes.

On the other hand, a HEVC-SCC fast coding device based on IPMS-CNN and spatial adjacent CU coding mode includes:

The data set production module creates and encodes video sequence data sets of different resolutions to obtain the real labels of whether each CU of HEVC-SCC uses IBC/PLT mode under different quantization parameters;

The network model building module builds a network model IPMS-CNN including an input layer, a feature extraction layer and an output expression layer; among them, three convolutional layers are built in the feature extraction layer to extract three types of feature maps;

The network model training module trains the constructed network model based on the produced data set and obtains the trained network model IPMS-CNN;

The network model prediction module inputs the CTU into the trained IPMS-CNN and obtains the mode prediction label to predict the mode selection of the CTU;

Current CU mode prediction module, calculates the number of IBC/PLT modes used by three adjacent CUs and the number of Intra modes used by three adjacent CUs , jointly predict the current 8×8CU mode based on the two quantitative relationships;

Encoding module, the encoder calls the prediction label based on the network model prediction step, and predicts the CU division result together with the current CU mode prediction step.

Compared with the prior art, the beneficial effects of the present invention are as follows:

(1) This invention first builds a database and trains the convolutional neural network model (IPMS-CNN) for IBC/PLT mode selection; secondly, the input CTU is passed through the mode selection network and the mode prediction label of the CTU is output; finally, the adjacent 3 The number of modes used by each CU is used to predict the mode selected by the current CU. This invention can reduce encoding time while maintaining encoding quality, and reduce the computational complexity of screen content video;

(2) The present invention adopts a four-scale feature fusion network structure. Among them, the down-sampled feature map will be sent to the connection layer together with the subsequent feature map obtained through the convolution layer. The down-sampled image provides a part of the deep layer. Features, and the feature map of the convolutional layer provides some shallow features. The combination of these shallow features and deep features can not only increase the amount of training data, but also provide more feature information to the fully connected layer, improving the model. Feature expression ability and accuracy of prediction model selection;

(3) The present invention adds the external vector QP to the fully connected layer of the output expression layer, which allows the model to better learn how to select the best encoding mode under different QPs, and allows the model to better adapt to various QPs. value, thereby generating a higher quality reconstructed video;

(4) The present invention combines the method of predicting large-size CU modes by convolutional neural network with the method of predicting small-size CU modes based on the number of modes used by spatially adjacent CUs, so that the mode of CUs can be predicted more accurately, thereby reducing mode selection. complexity.

Description of the drawings

Figure 1 is a flow chart of the HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode of the present invention;

Figure 2 is a schematic structural diagram of the IPMS-CNN convolutional neural network of the present invention;

Figure 3 is a schematic diagram of the current 8×8 CU and adjacent CUs of the present invention;

Figure 4 is a detailed flow chart of the present invention that connects the method of predicting large-sized CU patterns by a convolutional neural network and the method of predicting small-sized CU patterns based on the number of patterns used in spatially adjacent CUs;

Figure 5 is a schematic diagram of the MFF-CNN network structure of the present invention;

Figure 6 is a structural block diagram of the HEVC-SCC fast coding device based on IPMS-CNN and spatial adjacent CU coding mode of the present invention.

Detailed ways

The present invention will be further described below in conjunction with specific embodiments. It should be understood that these examples are only used to illustrate the invention and are not intended to limit the scope of the invention. In addition, it should be understood that after reading the teachings of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

In order to solve the problem of high CU division complexity in HEVC-SCC, this embodiment proposes a HEVC-SCC intra-frame CU fast division coding method based on multi-scale feature fusion (MFF-CNN), which is used to achieve high performance without affecting subjective quality. At the same time, it speeds up the encoding time and reduces the encoding complexity.

Specifically, see Figure 1, a HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode, including:

S101, the data set production step, establish and encode video sequence data sets of different resolutions, and obtain the real labels of whether each CU of HEVC-SCC uses IBC/PLT mode under different quantization parameters;

S102, network model construction step, construct a network model IPMS-CNN including an input layer, a feature extraction layer and an output expression layer; among them, three convolutional layers are built in the feature extraction layer, and three convolutional layers are built in the feature extraction layer to extract Three types of feature maps, plus the feature map obtained after downsampling, extract a total of four feature maps of different sizes;

S103, network model training step, based on the produced data set, train the constructed network model to obtain the trained IBC/PLT Mode Selection Convolution Neural Network (IPMS-CNN) model;

S104, input the LCU into the trained IPMS-CNN, and obtain the mode prediction labels of 64×64, 32×32, and 16×16CU to predict the mode selection of the CTU;

S105, 8×8 CU mode prediction step. The mode prediction of 8×8 CU predicts the mode that the current CU may select by counting the number of modes used by adjacent CUs in the spatial domain. Calculate the number of IBC/PLT modes used by three adjacent CUs and the number of Intra modes used by three adjacent CUs , jointly predict the current 8×8CU mode based on two quantitative relationships;

S106, encoding step, the encoder calls the prediction label based on the network model prediction step, and predicts the CU division result together with the current CU mode prediction step.

In this embodiment, the data set creation steps specifically include:

Create video sequence data sets of three different resolutions: 1024×576, 1792×102, and 2304×1280. The data sets cover image data sets and video data sets, and each resolution includes TGM/M, A, and CC. types of video sequences;

Encode all data set sequences through HM16.12+SCM8.3. Under the All Intra (AI) configuration, set the quantization parameters QP: 22, 27, 32, 37, and obtain each CU under the four QPs. The mode label of the IBC/PLT mode is used for the next step of network model training.

As shown in Figure 2, the IPMS-CNN network in this embodiment has three components: input layer, feature extraction layer and output expression layer. Three convolutional layers are built in the feature extraction layer of the network, three types of feature maps are extracted, and the downsampled feature maps are directly sent to the connection layer of the network. Therefore, the network can obtain feature maps of 12 different scales. The network uses the cross-entropy loss function as the objective function, and the formula is as follows:

in, Represents the cross entropy of the true value and the predicted value, , , Represents the real mode labels of the first-level (64×64), second-level (32×32), and third-level (16×16) CUs respectively. Represents the true mode tag of 64×64CTU, , It means four real mode labels of 32×32CTU, It means 4×4 real mode labels of 16×16CTU; similarly, , , Represents the prediction labels of the first level (64×64), the second level (32×32), and the third level (16×16) respectively. Represents the prediction mode label of 64×64CTU, , then represents four prediction mode labels of 32×32CTU, It represents 4 × 4 prediction mode labels of 16 × 16 CTU; the predicted labels and real labels of the network have been binarized, and the range is between [0, 1].

Since the quantization parameter QP is a parameter that can control the quality of video compression, the larger the QP, the higher the compression ratio, but the lower the quality of the compressed image. In HEVC-SCC, QP is also very important for the selection of SCC mode, because it can control the size of the quantization error, thus affecting the final image quality. When encoding the same video frame, the larger the QP, the rougher the encoded CU mode selection, and Intra mode is often selected; and the smaller the QP, the more likely the encoded QP will find a suitable reference and select IBC or PLT. model. Therefore, adding the external vector QP to the fully connected layer of the output expression layer can enable the model to better learn how to select the best encoding mode under different QP, allowing the model to better adapt to various QP values, and then Generate higher quality reconstructed videos.

In this embodiment, the network model finally outputs 21 binary labels indicating whether the 64×64, 32×32, and 16×16 CTUs will be divided and whether the IBC/PLT mode will be selected based on this.

The current CU mode prediction step specifically includes:

when When , the only candidate mode is Intra mode; when and When , the candidate modes are IBC and PLT modes; when and , the candidate modes are Intra, IBC and PLT modes.

Refer to Figure 3, which is a schematic diagram of the current 8×8 CU and adjacent CUs in this embodiment.

Refer to Figure 4, which is a detailed flowchart of this embodiment that connects the method of predicting large-sized CU patterns by a convolutional neural network and the method of predicting small-sized CU patterns based on the number of patterns used by spatially adjacent CUs. The details are as follows:

(a) The network inputs LCU, calls the multi-scale feature fusion convolutional neural network model MFF-CNN model (Multi-scale Feature Fusion Convolution Neural Network) (see Figure 5) and IPMS-CNN mode selection model, and outputs 21 binary values respectively. The label indicates whether the 64×64, 32×32, and 16×16 CTUs will be divided and whether the IBC/PLT mode will be selected based on this;

(b) When the CU size is 8×8, calculate the number of IBC, PLT modes and Intra modes used by three adjacent CUs. when When , the only candidate mode is Intra mode; when and When , the candidate modes are IBC and PLT modes; when and When , the candidate modes are Intra, IBC, and PLT modes;

(c) The encoder calls the network label according to step (a) and predicts the CU division result together with step (b), thereby skipping unnecessary mode traversal, reducing encoding time, and accelerating the encoding process of screen content video.

Refer to Figure 5, which is a schematic diagram of the MFF-CNN network structure in this embodiment.

Referring to Figure 6, this embodiment also discloses a HEVC-SCC fast coding device based on IPMS-CNN and spatial adjacent CU coding mode, including:

The data set production module 601 creates and encodes video sequence data sets of different resolutions to obtain the real labels of whether each CU of HEVC-SCC uses IBC/PLT mode under different quantization parameters;

The network model building module 602 constructs a network model IPMS-CNN including an input layer, a feature extraction layer and an output expression layer; among them, three convolutional layers are built in the feature extraction layer to extract three types of feature maps;

The network model training module 603 trains the constructed network model based on the produced data set and obtains the trained network model IPMS-CNN;

The network model prediction module 604 inputs the CTU into the trained IPMS-CNN and obtains the mode prediction label to predict the mode selection of the CTU;

The current CU mode prediction module 605 calculates the number of IBC/PLT modes used by three adjacent CUs. and the number of Intra modes used by three adjacent CUs , jointly predict the current 8×8CU mode based on the two quantitative relationships;

In the encoding module 606, the encoder calls the prediction tag based on the network model prediction step, and predicts the CU division result together with the current CU mode prediction step.

Specific implementation of each module of a HEVC-SCC fast encoding device based on IPMS-CNN and spatial adjacent CU coding mode. The same HEVC-SCC fast encoding method based on IPMS-CNN and spatial adjacent CU coding mode. This embodiment does not Repeat the explanation again.

The above are only specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantive changes to the present invention using this concept shall constitute an infringement of the protection scope of the present invention.

Claims (8)

1. A HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode, which is characterized by: In the data set production step, video sequence data sets of different resolutions are established and encoded to obtain the real labels of whether each CU of HEVC-SCC uses IBC/PLT mode under different quantization parameters; The network model construction step is to build a network model IPMS-CNN including an input layer, a feature extraction layer and an output expression layer. Among them, three convolutional layers are built in the feature extraction layer to extract three types of feature maps, plus the feature maps obtained after downsampling. , a total of four feature maps of different sizes are extracted; The network model training step is to train the constructed network model based on the produced data set to obtain the trained IBC/PLT mode selection convolutional neural network IPMS-CNN model; In the network model prediction step, the LCU is input into the trained IPMS-CNN to obtain the mode prediction label to predict the mode selection of the CTU; The current CU mode prediction step calculates the number of IBC/PLT modes used by three adjacent CUs NumSCC _Neigh and the number of Intra modes used by three adjacent CUs NumIntra _Neigh , and jointly predicts the current 8×8 CU mode based on the relationship between the two quantities; In the encoding step, the encoder calls the prediction label based on the network model prediction step, and predicts the CU division result together with the current CU mode prediction step; In the network model prediction step, the network model outputs 21 binary labels indicating whether the 64×64, 32×32, and 16×16 CTUs will be divided and whether the IBC/PLT mode will be selected on this basis; The current CU mode prediction step specifically includes: When the CU size is 8×8, calculate the number of IBC, PLT modes and Intra modes used by three adjacent CUs; specifically, when NumScc _Neigh = 0, the only candidate mode is the Intra mode; when NumSCC _Neigh ≠ 0 and When NumIntra _Neigh = 0, the candidate modes are IBC and PLT modes; when NumSCC _Neigh ≠0 and NumIntra _Neigh ≠0, the candidate modes are Intra, IBC and PLT modes. 2. The HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode according to claim 1, characterized in that the data set production step specifically includes: Self-made video sequence data sets of three different resolutions. The data sets cover image data sets and video data sets, including three types of video sequences: TGM/M, A, and CC; Then encode through the standard encoding software platform, and set the mode tag of the IBC/PLT mode of each CU under different quantization parameters QP under the full intra-frame configuration. 3. The HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode according to claim 1, characterized in that the data set includes: a training set, a verification set and a test set; Each set in the training set, validation set, and test set contains three subsets; the first subset has a resolution of 1024×576, the second subset has a resolution of 1792×1024, and the third subset has a resolution of 2304× 1280. 4. The HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode according to claim 1, characterized in that the quantization parameters include four quantization levels, respectively 22, 27, 32 and 37. 5. The HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode according to claim 1, characterized in that, in the network model building step, three convolutional layers are built in the feature extraction layer to extract Three types of feature maps, and the downsampled feature maps will be directly sent to the connection layer of the network. 6. The HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode according to claim 1, characterized in that the output expression layer includes a fully connected layer; characteristics of the fully connected layer The quantization parameter QP is added to the vector. 7. The HEVC-SCC fast coding method based on IPMS-CNN and spatial adjacent CU coding mode according to claim 1, characterized in that the loss function of the network model is as follows: Among them, H(*,*) represents the cross entropy of the true value and the predicted value, Respectively representing the real mode labels of the first level 64×64, the second level 32×32, and the third level 16×16CU, /> Represents the true mode tag of 64×64CTU, /> Represents four real mode tags of 32×32CTU,/> Represents 4×4 real mode tags of 16×16CTU; similarly, /> Represents the prediction labels of the first level 64×64, the second level 32×32, and the third level 16×16 respectively, /> Indicates the prediction mode label of 64×64CTU, /> It means four prediction mode labels of 32×32CTU,/> Represents 4 × 4 prediction mode labels of 16 × 16 CTU; both the prediction label and the real label of the network have been binarized, ranging from [0, 1]. 8. A HEVC-SCC fast coding device based on IPMS-CNN and spatial adjacent CU coding mode, which is characterized by including: The data set production module creates and encodes video sequence data sets of different resolutions to obtain the real labels of whether each CU of HEVC-SCC uses IBC/PLT mode under different quantization parameters; The network model building module builds a network model IPMS-CNN including an input layer, a feature extraction layer and an output expression layer; among them, three convolutional layers are built in the feature extraction layer to extract three types of feature maps; The network model training module trains the constructed network model based on the produced data set and obtains the trained network model IPMS-CNN; The network model prediction module inputs the CTU into the trained IPMS-CNN and obtains the mode prediction label to predict the mode selection of the CTU; The current CU mode prediction module calculates the number of IBC/PLT modes used by three adjacent CUs NumSCC _Neigh and the number of Intra modes used by three adjacent CUs NumIntra _Neight , and jointly predicts the current 8×8 CU mode based on the relationship between the two quantities; Coding module, the encoder calls the prediction label based on the network model prediction step, and predicts the CU division result together with the current CU mode prediction step; In the network model prediction module, the network model outputs 21 binary labels indicating whether the 64×64, 32×32, and 16×16 CTUs will be divided and whether the IBC/PLT mode will be selected on this basis; The current CU mode prediction module specifically includes: When the CU size is 8×8, calculate the number of IBC, PLT modes and Intra modes used by three adjacent CUs; specifically, when NumSCC _Neigh = 0, the only candidate mode is the Intra mode; when NumSCC _Neigh ≠ 0 and When NumINtra _Neigh = 0, the candidate modes are IBC and PLT modes; when NumSCC _Neigh ≠0 and NumIntra _Neigh ≠0, the candidate modes are Intra, IBC and PLT modes.