WO2022111233A1

WO2022111233A1 - Intra prediction mode coding method, and apparatus

Info

Publication number: WO2022111233A1
Application number: PCT/CN2021/128000
Authority: WO
Inventors: 杨海涛; 宋楠; 陈旭
Original assignee: 华为技术有限公司
Priority date: 2020-11-30
Filing date: 2021-11-01
Publication date: 2022-06-02
Also published as: CN114584776A

Abstract

Disclosed in the present application are an intra prediction mode coding method and a device. The present invention relates to the technical field of artificial intelligence (AI)-based video or image compression, and specifically relates to the technical field of neural network-based video compression. The method comprises an encoding method and a decoding method. The encoding method comprises: obtaining a probability vector for a current image block according to at least two among a residual block, a predicted block, and a reconstructed block of a peripheral image of the current image block as well as a prediction mode probability model; and encoding a target intra prediction mode into a code stream according to a target intra prediction mode of the current image block and the probability vector. The decoding method comprises: using a same method as an encoding end and obtaining a probability vector for a current image block; and determining a target intra prediction mode according to a code stream and the probability vector. When encoding or decoding is performed on a target intra prediction mode of a current image block, accuracy can be improved, storage overhead can be saved, and computational complexity can be reduced.

Description

Decoding method and apparatus for intra prediction mode

This application claims the priority of the Chinese patent application with the application number 202011387076.6 and the application title "Decoding Method and Device for Intra Prediction Mode", which was filed with the Chinese Patent Office on November 30, 2020, the entire contents of which are incorporated by reference in in this application.

technical field

The present application relates to the technical field of video or image compression based on artificial intelligence (AI), and in particular, to a decoding method and apparatus for intra prediction mode.

Background technique

Video coding (video encoding and decoding) is widely used in digital video applications such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat and video conferencing, DVD and Blu-ray discs, video content capture and editing systems And security applications for camcorders.

The large amount of video data that needs to be described even in the case of short films can create difficulties when the data is to be sent or otherwise transmitted over a network with limited bandwidth capacity. Therefore, video data is usually compressed before being transmitted in modern telecommunication networks. Since memory resources can be limited, the size of the video can also be an issue when storing the video on a storage device. Video compression devices typically use software and/or hardware on the source side to encode video data prior to transmission or storage, thereby reducing the amount of data required to represent digital video images. Then, the compressed data is received by the video decompression device at the destination side. With limited network resources and growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase compression ratios with little impact on image quality.

In recent years, the application of deep learning in the field of image and video encoding and decoding has gradually become a trend.

Among them, the decoding method of the intra-frame prediction mode of HEVC based on the neural network introduces the neural network model into the intra-frame prediction process. During encoding, the syntax elements related to the intra-frame prediction mode of the current image block are obtained by means of the neural network model, The acquired syntax elements are encoded; during decoding, the intra-frame prediction mode of the current image block is determined according to the syntax elements decoded in the code stream and the neural network model.

However, when encoding and decoding the intra prediction mode of the current image block in the prior art, there are problems such as limited accuracy, large storage overhead, and high computational complexity.

SUMMARY OF THE INVENTION

The present application provides a method and device for decoding an intra-frame prediction mode based on a neural network, which can improve the accuracy of encoding and decoding the intra-frame prediction mode of a current image block, save storage space, and reduce computational complexity. Thereby improving the encoding and decoding performance.

Specific embodiments are outlined in the appended independent claims, other embodiments are outlined in the dependent claims.

In a first aspect, the present application provides an intra-frame prediction mode encoding method, the method comprising: acquiring at least two of a reconstruction block, a prediction block and a residual block of a surrounding image block of a current image block; At least two of the reconstruction block, prediction block and residual block are combined with the prediction mode probability model to obtain the probability vector of the current image block. Multiple elements in the probability vector are in one-to-one correspondence with multiple intra prediction modes. Any element in the probability vector is in one-to-one correspondence. An element is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block; determine the target intra prediction mode of the current image block; The prediction mode is encoded into the codestream.

It can be seen that, in this embodiment of the present application, at least two of the reconstruction blocks, prediction blocks and residual blocks of the three surrounding image blocks of the current image block are obtained, because the reconstruction blocks, prediction blocks of each surrounding image block can be obtained through Two of the block and the residual block are calculated to obtain the other, so the embodiment of the present application can make full use of the relevant information of the surrounding image blocks, and then according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks The generated probability vector of the current image block is more accurate, which improves the accuracy of the data encoded in the code stream and improves the encoding performance.

In a possible design, the probability vector of the current image block is obtained according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks, the current image block and the prediction mode probability model, including: when the target frame When the intra prediction mode is the non-matrix weighted intra prediction MIP mode, the probability vector of the current image block is obtained according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks, the current image block and the prediction mode probability model.

It can be seen that when the target intra-frame prediction mode is a non-MIP mode, the method in the embodiment of the present application is used to encode the intra-frame prediction mode of the current image block, and the reconstructed blocks of the three surrounding image blocks of the current image block are obtained by obtaining the reconstructed blocks. , at least two of the prediction block and the residual block, the embodiments of the present application can make full use of the relevant information of the surrounding image blocks, and then generate a The probability vector of the current image block is more accurate, thereby improving the accuracy of the data encoded in the code stream and improving the encoding performance when the target intra-frame prediction mode is a non-MIP mode.

In a possible design, the probability vector of the current image block is obtained according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks and the prediction mode probability model, including: the reconstruction block of the surrounding image blocks The first data block is obtained by splicing or concatenating at least two of the prediction block and the residual block; the probability vector of the current image block is obtained according to the first data block and the prediction mode probability model.

It should be understood that the prediction mode probability model in this embodiment of the present application may be a neural network model, and second data blocks of different sizes correspond to different neural network models.

It can be seen that in this embodiment of the present application, since at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks can fully characterize the relevant information of the surrounding image blocks, , at least two of the prediction block and the residual block are spliced or concatenated to obtain the first data block, so the first data block can fully reflect the relevant information of the current image block, and then the current image block obtained according to the first data block. The probability vector of the image block is more accurate, thereby improving the accuracy of the data encoded in the code stream and improving the encoding performance.

In a possible design, obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model includes: when the size of the first data block is the target size, inputting the first data block into the prediction mode probability The model is processed to obtain a probability vector for the current image patch, or;

When the size of the first data block is not the target size, a size transformation operation is performed on the first data block to obtain a second data block, where the size of the second data block is equal to the target size, and the size transformation operation includes scaling and transposing. In at least one item, scaling includes horizontal scaling, vertical scaling, or equal scaling in both horizontal and vertical directions; inputting the second data block into a prediction mode probability model for processing to obtain a probability vector of the current image block.

In a possible design, when the size of the first data block is not the target size, performing a size transformation operation on the first data block to obtain the second data block may include four methods: (1) when the first data block is not the target size When the size of a data block is not the target size, and the ratio of the first side length and the second side length of the first data block is equal to the ratio of the first side length and the second side length of the target size, the first data block is processed Proportional scaling in the horizontal and vertical directions to obtain the second data block; (2) when the size of the first data block is not the target size, and the ratio of the first side length to the second side length of the first data block When it is equal to the ratio of the second side length to the first side length of the target size, transpose the first data block, and proportionally scale the horizontal and vertical directions to obtain the second data block; (3) When the first data block is obtained; When the size of a data block is not the target size, and the first variable length and the second side length of the first data block are respectively equal to the second side length and the first side length of the target size, the first data block is transposed, Obtain the second data block; (4) when the size of the first data block is not the target size, scale the first data block to obtain the second data block.

It should be understood that the target sizes used in the above-mentioned four manners in this embodiment of the present application may be different, and the target size may be one or more. At the same time, the number of target sizes is smaller than the number of first data block sizes; the above-mentioned first side length and the second side lengths can be width and height respectively, or height and width respectively; in addition, the transposition and scaling operations performed on the first data block in the above-mentioned second method have no sequence, and the transposition is performed first and then the scaling is performed. The size of the second data block obtained in the two methods is the same as that of scaling first and then transposing.

It can be seen that, in the above embodiment, since there are various sizes of the first data block, by performing a size transformation operation on the first data block to obtain the second data block, the number of sizes of the second data block can be effectively reduced, Thus, the number of neural network models corresponding to the second data blocks of different sizes is reduced, thereby saving the storage space of the neural network model and improving the coding performance. After the transformation operation is reduced to a second data block with a smaller size, the computational complexity of the subsequent calculation of the probability vector by using the neural network model can be effectively reduced, thereby improving the coding efficiency.

When the size of the first data block is not the target size, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, for the first data block A size transformation operation is performed to obtain a second data block, the size of the second data block is the target size, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or horizontal and vertical scaling Proportional scaling in two directions; input the second data block into the prediction mode probability model for processing to obtain the probability vector of the current image block.

It can be seen that, in the embodiment of the present application, the process of determining the second data block is based on the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, The self-information of the current image block is fully utilized, and the correlation between the second data block and the current image block is enhanced, thereby improving the accuracy of the probability vector obtained from the second data block, thereby improving the accuracy of the data in the code stream. , which improves encoding performance.

In a possible design, the size transformation operation includes scaling, the above-mentioned size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, to the first data block. The data block is subjected to a size transformation operation to obtain a second data block, including: when the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the absolute value of the horizontal gradient is the same as the vertical gradient. When the sum of the absolute values of , is less than the preset threshold, the first data block is scaled to obtain a second data block, and the first and second side lengths of the second data block are 4M/N, 4 respectively; When the first side length M of a data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the preset threshold, the first data block Scaling is performed to obtain a second data block, and the first side length and the second side length of the second data block are 8M/N, 8 respectively; when the first side length M of the first data block is smaller than the first data block. When the length of the two sides is N, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset threshold, the first data block is scaled to obtain a second data block. The sum of the first side lengths of the second data block The second side lengths are respectively 4 and 4N/M; when the first side length M of the first data block is less than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the preset threshold, the first data block is scaled to obtain a second data block, and the first side length and the second side length of the second data block are respectively 8, 8N/M; wherein, M and N are positive integer.

It can be seen that, in the embodiment of the present application, the process of determining the second data block according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, is sufficient The self-information of the current image is used, thereby improving the accuracy of the probability vector obtained from the second data block subsequently, thereby improving the accuracy of the data in the code stream; at the same time, the number of size types of the second data block can be reduced by scaling , thereby also reducing the number of neural network models corresponding to second data blocks of different sizes, saving storage space for storing neural network models, and improving coding performance.

In a possible design, the size transformation operation includes at least one of scaling and transposition, which is based on the size relationship between the first side length and the second side length of the first data block, and/or the level of the current image block. Gradient and vertical gradient, performing a size transformation operation on the first data block to obtain a second data block, including: when the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset threshold, the first data block is scaled to obtain the second data block, and the first and second side lengths of the second data block are respectively 4M/N, 4; when the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the preset At the threshold, the first data block is scaled to obtain a second data block, and the first side length and the second side length of the second data block are 8M/N, 8 respectively;

When the first side length M of the first data block is smaller than the second side length N of the first data block, perform the following operations on the first data block to obtain the second data block: when the absolute value of the horizontal gradient of the first data block is When the sum of the absolute value of the value and the vertical gradient is less than the preset threshold, transpose and scale the first data block to obtain a second data block, and the first and second side lengths of the second data block are 4N respectively. /M, 4; when the sum of the absolute value of the horizontal gradient of the first data block and the absolute value of the vertical gradient is greater than or equal to the preset threshold, transpose and scale the first data block to obtain the second data block, The length of the first side and the length of the second side of the second data block are respectively 8N/M, 8; wherein, M and N are positive integers.

It can be seen that, in the embodiment of the present application, the process of determining the second data block according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, The self-information of the current image is fully utilized to determine the size of the second data block, and the correlation between the second data block and the current image block is enhanced, thereby improving the accuracy of the probability vector obtained according to the second data block subsequently, thereby improving the At the same time, by performing scaling and transposing operations, the number of second data block size types can be reduced, and the number of neural network models corresponding to second data blocks of different sizes can also be reduced. , thereby saving the storage space for storing the neural network model and improving the coding performance.

In a possible design, the above-mentioned encoding the target intra prediction mode into the code stream according to the target intra prediction mode and the probability vector includes: when the size transformation operation does not include transposition, according to the first intra prediction mode of the target intra prediction mode. an identifier and a probability vector, determine the first reference value, and encode the first reference value to obtain the code stream corresponding to the target intra-frame prediction mode; when the size transformation operation includes transposition, and the target intra-frame prediction mode is the angle prediction mode , determine the second identifier according to the preset constant and the first identifier of the target intra-frame prediction mode, determine the second reference value according to the second identifier and the probability vector; encode the second reference value to obtain the target intra-frame prediction mode Corresponding code stream; when the size transformation operation includes transposition, and the target intra-frame prediction mode is a non-angle prediction mode, determine the first reference value according to the first identifier and probability vector of the target intra-frame prediction mode; for the first reference The value is encoded to obtain the code stream corresponding to the target intra prediction mode.

It should be understood that, in this embodiment of the present application, the preset constant is equal to the number of intra-frame prediction modes in the multi-function video coding VVC technology, that is, 67; the first identifier may be the mode value corresponding to the target intra-frame prediction mode; the angle prediction mode It is 65 intra-frame prediction modes with a mode value of 2-67 in VVC, and the non-angle prediction mode is a planar prediction mode with a mode value of 0 and a DC-DC prediction mode with a mode value of 1.

It can be seen that, in the embodiment of the present application, when the size transformation operation includes transposition and the target intra-frame prediction mode is the angle prediction mode, the first identifier of the target intra-frame prediction mode is processed by a preset constant to obtain the first identifier of the target intra-frame prediction mode. Second identification, and then determine the second reference value according to the second identification, to ensure that when the first data block is transposed, the corresponding correct second reference value is programmed into the code stream, thereby ensuring the correctness of the encoding process.

In a second aspect, the present application provides an intra-frame prediction mode encoding method, the method comprising: acquiring surrounding image blocks of a current image block; splicing or concatenating the surrounding image blocks to obtain a first data block; The block performs a size transformation operation to obtain the second data block, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling or proportional scaling in both horizontal and vertical directions; Input the second data block into the prediction mode probability model for processing to obtain the probability vector of the current image block. Multiple elements in the probability vector correspond to multiple intra prediction modes one-to-one, and any element in the probability vector is used to represent Use the probability of the intra prediction mode corresponding to any element when predicting the current image block; determine the target intra prediction mode of the current image block; encode the target intra prediction mode into the code according to the target intra prediction mode and the probability vector flow.

It can be seen that, in the embodiment of the present application, by scaling the first data block to obtain a second data block with a smaller size, the computational complexity of the subsequent use of the second data block to participate in the calculation of the probability vector can be effectively reduced, so that Improve coding efficiency; at the same time, through the size transformation operation, the number of size types of the second data block can be reduced, thereby reducing the number of neural network models corresponding to second data blocks of different sizes, thereby saving the storage space of the neural network model. encoding performance.

In a possible design, performing a size transformation operation on the first data block to obtain the second data block includes: when the target intra-frame prediction mode is a non-matrix-weighted intra-frame prediction MIP mode, and the first data block is When the size is not the target size, a size transformation operation is performed on the first data block to obtain a second data block, and the size of the second data block is equal to the target size.

It should be understood that the target size may be set according to actual scenarios, and the number of target sizes is smaller than the number of first data block sizes.

It can be seen that, in this embodiment of the present application, since there are various sizes of the first data block, by performing a size transformation operation on the first data block to obtain the second data block, the number of sizes of the second data block can be effectively reduced , thereby reducing the number of neural network models corresponding to second data blocks of different sizes when the target intra prediction mode is the non-matrix weighted intra prediction MIP mode, thereby saving the storage space of the neural network model and improving the coding performance At the same time, by scaling the first data block, a second data block with a smaller size is obtained, so that when the target intra-frame prediction mode is a non-matrix weighted intra-frame prediction MIP mode, the subsequent use of the second data block can be effectively reduced. The computational complexity when the block participates in the calculation of the probability vector, thereby improving the coding efficiency.

In a possible design, performing a size transformation operation on the first data block to obtain the second data block includes: when the target intra-frame prediction mode is a non-matrix-weighted intra-frame prediction MIP mode, and the first data block is When the size is not the target size, according to the size relationship between the length of the first side and the length of the second side of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, perform a size transformation operation on the first data block to A second data block is obtained, and the size of the second data block is equal to the target size.

It should be understood that the above-mentioned first side length and second side length may be width and height, respectively, or height and width, respectively.

In a possible design, when the target intra-frame prediction mode is a non-matrix-weighted intra-frame prediction MIP mode, and the size of the first data block is equal to the target size, the first data block is input into the prediction mode probability model for processing to obtain Get the probability vector of the current image patch.

It can be seen that, in the embodiment of the present application, when the size of the first data block is equal to the set target size, the size transformation operation is not performed, and the first data block is directly used to participate in the generation of the probability vector, which simplifies the process of analyzing the target frame. The process of coding in prediction mode improves coding efficiency.

In a possible design, the size transformation operation includes at least one of scaling and transposition; the above is based on the size relationship between the length of the first side and the length of the second side of the first data block, and/or the level of the current image block. Gradient and vertical gradient, performing a size transformation operation on the first data block to obtain a second data block, including: when the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset threshold, the first data block is scaled to obtain the second data block, and the first and second side lengths of the second data block are respectively 4M/N, 4; when the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the preset At the threshold, the first data block is scaled to obtain a second data block, and the first side length and the second side length of the second data block are 8M/N, 8 respectively;

In a possible design, the above-mentioned encoding the target intra prediction mode into the code stream according to the target intra prediction mode and the probability vector includes: when the size transformation operation does not include transposition, according to the first intra prediction mode of the target intra prediction mode. an identifier and a probability vector to determine the first reference value; the first reference value is encoded to obtain a code stream corresponding to the target intra-frame prediction mode; when the size transformation operation includes transposition, and the target intra-frame prediction mode is the angle prediction mode , determine the second identifier according to the preset constant and the first identifier of the target intra-frame prediction mode, determine the second reference value according to the second identifier and the probability vector; encode the second reference value to obtain the target intra-frame prediction mode Corresponding code stream; when the size transformation operation includes transposition, and the target intra-frame prediction mode is a non-angle prediction mode, determine the first reference value according to the first identifier and probability vector of the target intra-frame prediction mode; for the first reference The value is encoded to obtain the code stream corresponding to the target intra prediction mode.

In a third aspect, the present application provides a decoding method for an intra-frame prediction mode, the method comprising: acquiring a code stream corresponding to a current image block, and a reconstruction block, a prediction block, and a residual block of the surrounding image blocks of the current image block. At least two; obtain a probability vector of the current image block according to at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks and the prediction mode probability model, and multiple elements in the probability vector are related to multiple intra prediction modes. One-to-one correspondence, any element in the probability vector is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block; according to the code stream and the probability vector, determine the target intra prediction of the current image block mode, and determine the prediction block of the current image block according to the target intra prediction mode.

It can be seen that, in this embodiment of the present application, at least two of the reconstruction blocks, prediction blocks and residual blocks of the three surrounding image blocks of the current image block are obtained, because the reconstruction blocks, prediction blocks of each surrounding image block can be obtained through Two of the block and the residual block are calculated to obtain the other, so the embodiment of the present application can make full use of the relevant information of the surrounding image blocks, and then according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks The generated probability vector of the current image block is more accurate, and subsequently, the target intra prediction mode of the current image block can be accurately determined according to the probability vector and the code stream, thereby improving the decoding performance.

It can be seen that in this embodiment of the present application, since at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks can fully characterize the relevant information of the surrounding image blocks, , at least two of the prediction block and the residual block are spliced or cascaded to obtain the first data block, so the first data block can fully reflect the relevant information of the current image block, and the current image obtained according to the first data block The probability vector of the block is also more accurate, and subsequently, the target intra prediction mode of the current image block can be accurately determined according to the probability vector and the code stream, thereby improving the decoding performance.

In a possible design, obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model includes: when the size of the first data block is the target size, inputting the first data block into the prediction mode probability The model is processed to obtain the probability vector of the current image block, or; when the size of the first data block is not the target size, a size transformation operation is performed on the first data block to obtain the second data block, the The size is equal to the target size, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or proportional scaling in both horizontal and vertical directions; inputting the second data block into the prediction mode The probability model is processed to obtain the probability vector of the current image patch.

It should be understood that the specific implementation process of each step in the embodiment of the present application is the same as the corresponding step in the first aspect, and details are not repeated here.

It can be seen that, in this embodiment of the present application, since there are various sizes of the first data block, by performing a size transformation operation on the first data block to obtain the second data block, the number of sizes of the second data block can be effectively reduced , thereby reducing the number of neural network models corresponding to the second data blocks of different sizes, thereby saving the storage space of the neural network model and improving the decoding performance. In addition, for the first data block with simpler texture and larger size, after After the size transformation operation is reduced to a second data block with a smaller size, the computational complexity of the subsequent calculation of the probability vector by using the neural network model can be effectively reduced, thereby improving the decoding efficiency.

When the size of the first data block is not the target size, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, for the first data block A size transformation operation is performed to obtain a second data block, the first side length and the second side length are the lengths of two mutually perpendicular sides in the first data block, the size of the second data block is the target size, and the size transformation operation includes at least one of scaling and transposition, the scaling includes horizontal scaling, vertical scaling or equal scaling in both horizontal and vertical directions; inputting the second data block into the prediction mode probability model for processing to obtain the current image The probability vector for the block.

It can be seen that, in the embodiment of the present application, the process of determining the second data block is based on the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, The self-information of the current image block is fully utilized, and the correlation between the second data block and the current image block is enhanced, thereby improving the accuracy of the probability vector obtained according to the second data block, so that the probability vector and the code can be obtained later. Streaming accurately determines the target intra prediction mode for the current image block, improving decoding performance.

It can be seen that, in the embodiment of the present application, the process of determining the second data block according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, is sufficient The self-information of the current image is utilized, thereby improving the accuracy of the probability vector obtained subsequently according to the second data block, so that the target intra prediction mode of the current image block subsequently determined according to the probability vector and the code stream is more accurate; , the number of size types of the second data block can be reduced by scaling, thereby reducing the number of neural network models corresponding to the second data blocks of different sizes, saving the storage space for storing the neural network model and improving the decoding performance. .

It can be seen that, in the above embodiment, the process of determining the second data block according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, is sufficient. The self-information of the current image is used to determine the size of the second data block, which enhances the correlation between the second data block and the current image block, thereby improving the accuracy of the subsequent probability vector obtained according to the second data block, thereby improving the accuracy of the data in the code stream; at the same time, by performing scaling and transposing operations, the number of sizes of second data blocks can be reduced, and the number of neural network models corresponding to second data blocks of different sizes can also be reduced. Thus, the storage space for storing the neural network model is saved, and the coding performance is improved.

In a possible design, the above-mentioned determining the target intra prediction mode of the current image block according to the code stream and the probability vector includes: decoding the code stream to obtain the target reference value; when the size transformation operation does not include transposition, Determine the target intra-frame prediction mode according to the target reference value and the probability vector; when the size transformation operation includes transposition, determine the first mark according to the target reference value and the probability vector; When the intra-frame prediction mode corresponding to the first mark is the angle prediction mode , determine the target intra-frame prediction mode according to the preset constant and the first identifier; when the intra-frame prediction mode corresponding to the first identifier is a non-angle prediction mode, determine the target intra-frame prediction mode according to the target reference value and the probability vector.

It can be seen that, in the embodiment of the present application, when the size transformation operation performed on the first data block includes transposition, and the intra-frame prediction mode corresponding to the above-mentioned first identifier is an angle prediction mode, it indicates that the above-mentioned first data block is in the encoding process. An identifier has undergone transposition processing. At this time, the first identifier is transposed according to a preset constant to obtain a second identifier. The intra-frame prediction mode corresponding to the second identifier is the target intra-frame prediction mode of the current image block. In the above manner, it can be ensured that after the first data block is transposed, the correct target intra-frame prediction mode is obtained in the decoding process, and then the real prediction block of the current image block is obtained according to the target intra-frame prediction mode.

In a fourth aspect, the present application provides a decoding method for an intra prediction mode, the method comprising: acquiring a code stream corresponding to a current image block and surrounding image blocks of the current image block; splicing or concatenating the surrounding image blocks to obtain The first data block; a size transformation operation is performed on the first data block to obtain a second data block, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or horizontal and vertical scaling. Proportional scaling in two directions; input the second data block into the prediction mode probability model for processing to obtain the probability vector of the current image block, multiple elements in the probability vector correspond to multiple intra prediction modes one-to-one, and the probability Any element in the vector is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block; according to the code stream and the probability vector, determine the target intra prediction mode of the current image block, and according to the target Intra prediction mode, which determines the prediction block of the current image block.

It can be seen that, in the embodiment of the present application, for a first data block with a simpler texture and a larger size, after the size transformation operation is reduced to a second data block with a smaller size, the subsequent use of the neural network model can be effectively reduced. The computational complexity when calculating the probability vector improves the decoding efficiency; in addition, since there are various sizes of the first data block, the second data block can be obtained by performing a size transformation operation on the first data block, which can effectively reduce the second data block. The number of size types is reduced, thereby reducing the number of neural network models corresponding to second data blocks of different sizes, thereby saving the storage space of the neural network model and improving the decoding performance.

In a possible design, performing the size transformation operation on the first data block to obtain the second data block includes: when the size of the first data block is not the target size, performing a size transformation operation on the first data block , to obtain a second data block whose size is equal to the target size.

It can be seen that in the embodiment of the present application, the target size can be set according to the actual application scenario, and changing the size of the first data block to the target size through the size transformation operation can effectively meet the actual scene requirements; at the same time, because the target size The number of types is smaller than the number of types of the size of the first data block, so after the size transformation operation, the number of types of the size of the second data block can be effectively reduced, and the number of neural network models corresponding to the second data blocks of different sizes can be reduced. Save the memory space of the neural network model and improve the decoding performance.

In a possible design, performing the size transformation operation on the first data block to obtain the second data block includes: when the size of the first data block is not the target size, according to the first side of the first data block The size relationship between the length and the second side length, and/or the horizontal gradient and vertical gradient of the current image block, perform a size transformation operation on the first data block to obtain a second data block, and the size of the second data block is equal to the target size.

In a possible design, the above method further includes: when the size of the first data block is equal to the target size, inputting the first data block into the prediction mode probability model for processing to obtain a probability vector of the current image block.

It can be seen that in this embodiment of the present application, when the size of the first data block is equal to the set target size, the size transformation operation is not performed, and the first data block is directly used to participate in the generation of the probability vector, which simplifies the probability of the current image block. The vector generation process improves the decoding efficiency.

It should be understood that the specific process and beneficial effects of scaling the first data block to obtain the second data block are the same as the corresponding process in the third aspect, and will not be repeated here.

It should be understood that the above process and beneficial effects of determining the target intra prediction mode of the current image block according to the code stream and the probability vector are the same as the corresponding process in the third aspect, and will not be repeated here.

In a fifth aspect, the present application provides an encoding device, the beneficial effects of which can be referred to the description of the first aspect, and are not repeated here. The encoding device has the function of implementing the behavior in the method example of the first aspect above. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the encoding device includes: an obtaining unit, configured to obtain at least two of a reconstruction block, a prediction block and a residual block of surrounding image blocks of the current image block; At least two of the reconstructed block, prediction block and residual block of the block and the prediction mode probability model obtain the probability vector of the current image block, and multiple elements in the probability vector correspond to multiple intra prediction modes one-to-one. Any element of is used to characterize the probability of adopting the intra prediction mode corresponding to any element when predicting the current image block; the determination unit is used to determine the target intra prediction mode of the current image block; the coding unit is used to determine the target intra prediction mode according to the target Intra prediction mode and probability vector, encoding the target intra prediction mode into the code stream. These modules can perform the corresponding functions in the method examples of the first aspect. For details, please refer to the detailed descriptions in the method examples, which will not be repeated here.

In a sixth aspect, the present application provides an encoding apparatus, the beneficial effects of which can be referred to the description of the second aspect, and are not repeated here. The encoding device has the function of implementing the behavior in the method example of the second aspect above. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the encoding device includes: an acquisition unit, used for acquiring surrounding image blocks of the current image block; a processing unit, used for splicing or concatenating the surrounding image blocks to obtain a first data block; A data block is subjected to a size transformation operation to obtain a second data block. The size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or an equal ratio between horizontal and vertical directions. scaling; and for processing the second data block into the prediction mode probability model to obtain a probability vector of the current image block, where multiple elements in the probability vector correspond to multiple intra prediction modes one-to-one, and any element in the probability vector is in one-to-one correspondence. An element is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block; the determining unit is used to determine the target intra prediction mode of the current image block; the coding unit is used to determine the target intra prediction mode according to the target intra frame Prediction mode and probability vector, encoding the target intra prediction mode into the code stream. These modules can perform the corresponding functions in the method examples of the second aspect. For details, please refer to the detailed descriptions in the method examples, which will not be repeated here.

In a seventh aspect, the present application provides a decoding apparatus. For beneficial effects, reference may be made to the description of the third aspect, which will not be repeated here. The decoding device has a function to implement the behavior in the method example of the third aspect above. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the decoding apparatus includes: an obtaining unit, configured to obtain a code stream corresponding to the current image block, and at least two of the reconstruction blocks, prediction blocks and residual blocks of surrounding image blocks of the current image block A processing unit for obtaining the probability vector of the current image block according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image block and the prediction mode probability model, and multiple elements in the probability vector and multiple intraframes The prediction modes are in one-to-one correspondence, and any element in the probability vector is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block; the decoding unit is used to determine the current image block according to the code stream and the probability vector. The target intra prediction mode of the image block, and the prediction block of the current image block is determined according to the target intra prediction mode. These modules can perform the corresponding functions in the method examples of the third aspect. For details, please refer to the detailed descriptions in the method examples, which will not be repeated here.

In an eighth aspect, the present application provides a decoding apparatus. For beneficial effects, reference may be made to the description of the fourth aspect, which will not be repeated here. The decoding device has a function to implement the behavior in the method example of the fourth aspect above. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the decoding device includes: an acquiring unit, configured to acquire a code stream corresponding to the current image block and surrounding image blocks of the current image block; a processing unit, configured to splicing or concatenating the surrounding image blocks to obtain a first data block; perform a size transformation operation on the first data block to obtain a second data block, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling or horizontal scaling and proportional scaling in two vertical directions; and for processing the second data block into the prediction mode probability model to obtain a probability vector of the current image block, multiple elements in the probability vector and multiple intra prediction modes One-to-one correspondence, any element in the probability vector is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block; the decoding unit is used to determine the current image block according to the code stream and the probability vector The target intra-frame prediction mode is determined, and the prediction block of the current image block is determined according to the target intra-frame prediction mode. These modules can perform the corresponding functions in the method example of the fourth aspect. For details, please refer to the detailed description in the method example, which will not be repeated here.

The method described in the first aspect of the present application may be performed by the apparatus described in the fifth aspect of the present application. Other features and implementations of the method described in the first aspect of the present application directly depend on the functionality and implementation of the device described in the fifth aspect of the present application.

The method described in the second aspect of the present application may be performed by the apparatus described in the sixth aspect of the present application. Other features and implementations of the method described in the second aspect of the present application directly depend on the functionality and implementation of the device described in the sixth aspect of the present application.

The method described in the third aspect of the present application may be performed by the apparatus described in the seventh aspect of the present application. Other features and implementations of the method described in the third aspect of the present application directly depend on the functionality and implementation of the device described in the seventh aspect of the present application.

The method described in the fourth aspect of the present application may be performed by the apparatus described in the eighth aspect of the present application. Other features and implementations of the method described in the fourth aspect of the present application directly depend on the functionality and implementation of the apparatus described in the eighth aspect of the present application.

In a ninth aspect, the present application provides an encoder (20) comprising a processing circuit, the encoder being operable to perform the method in all or any of the possible embodiments of the first aspect and the second aspect.

In a tenth aspect, the present application provides a decoder (30) comprising a processing circuit, the decoder being operable to perform the method in all or any of the possible embodiments of the third aspect and the fourth aspect.

In an eleventh aspect, the present application provides an encoder, comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the processing and stores a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform all or any of the possible embodiments of the first aspect or the second aspect above Methods.

In a twelfth aspect, the present application provides a decoder, comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing data from the A program executed by the processor, wherein the program, when executed by the processor, causes the decoder to execute the method in all or any of the possible embodiments of the third aspect or the fourth aspect.

In a thirteenth aspect, the present application provides a non-transitory computer-readable storage medium comprising program code that, when executed by a computer device, executes all of the above-mentioned first, second, third and fourth aspects or the method in any possible embodiment.

In a fourteenth aspect, the present application provides a non-transitory computer-readable storage medium, comprising a bitstream encoded according to the method in all or any of the possible embodiments of the first aspect or the second aspect.

In a fifteenth aspect, the present application provides a computer program product, comprising program code, the program code executing all or any of the above-mentioned first, second, third and fourth aspects when run on a computer or processor method in one possible embodiment.

One or more embodiments are described in detail in the accompanying drawings and the description below. Other features, objects, and advantages are apparent from the description, drawings, and claims.

Description of drawings

The accompanying drawings used in the embodiments of the present application will be introduced below.

1A is a block diagram of an example video coding system for implementing embodiments of the present application, wherein the system utilizes a neural network to encode or decode video images;

1B is a block diagram of another example of a video coding system for implementing embodiments of the present application, wherein the video encoder and/or video decoder use a neural network to encode or decode video images;

2 is a block diagram of an example example of a video encoder for implementing embodiments of the present application, wherein the video encoder 20 uses a neural network to encode video images;

3 is a block diagram of an example example of a video decoder for implementing embodiments of the present application, wherein the video decoder 30 uses a neural network to decode video images;

4 is a schematic block diagram of a video decoding apparatus for implementing an embodiment of the present application;

5 is a schematic block diagram of a video decoding apparatus for implementing an embodiment of the present application;

6 is a schematic diagram of a neural network structure for obtaining a probability vector of a current image block according to an embodiment of the application;

7 is a flowchart of a method for encoding a target intra prediction mode of a current image block in an embodiment of the application;

8 is a flowchart of another method for encoding a target intra prediction mode of a current image block in an embodiment of the application;

9 is a schematic diagram of multiple intra prediction modes in an embodiment of the present application;

10 is a schematic diagram of a splicing method of a current image block in an embodiment of the present application;

Figures 11-1 to 11-4 are schematic diagrams showing four kinds of positional relationships of the current image block in the current frame in the embodiment of the present application;

12 is a flowchart of an encoder encoding related syntax elements of an intra prediction mode of a current image block according to an embodiment of the present application;

13 is a flowchart of a method for decoding a target intra prediction mode of a current image block in an embodiment of the present application;

14 is a flowchart of another method for decoding a target intra prediction mode of a current image block in an embodiment of the present application;

15 is a schematic block diagram of an encoding apparatus in an embodiment of the present application;

FIG. 16 is a schematic block diagram of a decoding apparatus in an embodiment of the present application.

Detailed ways

Embodiments of the present application provide an AI-based video image compression technology, and in particular, provide a neural network-based intra-frame prediction mode encoding and decoding technology to improve traditional hybrid video encoding and decoding systems.

Video coding generally refers to the processing of sequences of images that form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" may be used as synonyms. Video encoding (or commonly referred to as encoding) includes two parts: video encoding and video decoding. Video encoding is performed on the source side and typically involves processing (eg, compressing) the original video image to reduce the amount of data required to represent the video image (and thus store and/or transmit more efficiently). Video decoding is performed on the destination side and typically involves inverse processing relative to the encoder to reconstruct the video image. The "encoding" of a video image (or commonly referred to as an image) in relation to the embodiments should be understood as the "encoding" or "decoding" of a video image or a video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding, CODEC).

In the case of lossless video coding, the original video image can be reconstructed, ie the reconstructed video image has the same quality as the original video image (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed through quantization, etc. to reduce the amount of data required to represent the video image, and the decoder side cannot completely reconstruct the video image, that is, the quality of the reconstructed video image is higher than that of the original video image. low or poor.

Several video coding standards fall under the category of "lossy hybrid video codecs" (ie, combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is usually partitioned into sets of non-overlapping blocks, usually encoded at the block level. In other words, encoders typically process i.e. encode video at the block (video block) level, eg, by spatial (intra) prediction and temporal (inter) prediction to generate prediction blocks; block) to subtract the prediction block to get the residual block; transform the residual block in the transform domain and quantize the residual block to reduce the amount of data to be transmitted (compressed), and the decoder side will process inversely with respect to the encoder Partially applied to encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder needs to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (eg, intra- and inter-prediction) and/or reconstructed pixels for processing, ie, encoding subsequent blocks.

In the following embodiments of the decoding system 10, the encoder 20 and the decoder 30 are described with respect to FIGS. 1A-3.

1A is a schematic block diagram of an exemplary coding system 10, such as a video coding system 10 (or simply coding system 10) that may utilize the techniques of this application. Video encoder 20 (or encoder 20 for short) and video decoder 30 (or decoder 30 for short) in video coding system 10 represent devices, etc. that may be used to perform techniques in accordance with the various examples described in this application .

As shown in FIG. 1A , the decoding system 10 includes a source device 12 for providing encoded image data 21 such as encoded images to a destination device 14 for decoding the encoded image data 21 .

The source device 12 includes an encoder 20 and, alternatively, an image source 16 , a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22 .

Image source 16 may include or be any type of image capture device for capturing real-world images, etc., and/or any type of image generation device, such as a computer graphics processor or any type of user for generating computer animation images. Devices used to acquire and/or provide real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and/or any combination thereof (e.g., augmented reality, AR) images). The image source may be any type of memory or storage that stores any of the above-mentioned images.

To distinguish the processing performed by the preprocessor (or preprocessing unit) 18 , the image (or image data) 17 may also be referred to as the original image (or original image data) 17 .

The preprocessor 18 is used to receive the (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image (or preprocessed image data) 19 . For example, the preprocessing performed by the preprocessor 18 may include trimming, color format conversion (eg, from RGB to YCbCr), toning, or denoising. It is understood that the preprocessing unit 18 may be an optional component.

A video encoder (or encoder) 20 is used to receive preprocessed image data 19 and to provide encoded image data 21 (described further below with respect to FIG. 2 etc.).

The communication interface 22 in the source device 12 can be used to: receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) over the communication channel 13 to another device such as the destination device 14 or any other device for storage or rebuild directly.

The destination device 14 includes a decoder 30 and may additionally, alternatively, include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 and a display device 34 .

The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, The encoded image data 21 is supplied to the decoder 30 .

Communication interface 22 and communication interface 28 may be used through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, etc., or through any type of network, such as a wired network, a wireless network, or any Combination, any type of private network and public network, or any type of combination, send or receive encoded image data (or encoded data) 21 .

For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and/or use any type of transfer encoding or processing to process the encoded image data for transmission over a communication link or communication network transfer on.

The communication interface 28 corresponds to the communication interface 22 and may be used, for example, to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and/or decapsulation to obtain encoded image data 21 .

Both the communication interface 22 and the communication interface 28 can be configured as a one-way communication interface as indicated by the arrows from the source device 12 to the corresponding communication channel 13 of the destination device 14 in FIG. 1A, or a two-way communication interface, and can be used to send and receive messages etc. to establish a connection, acknowledge and exchange any other information related to a communication link and/or data transfer such as encoded image data transfer, etc.

A video decoder (or decoder) 30 is used to receive encoded image data 21 and to provide decoded image data (or decoded image data) 31 (described further below with reference to FIG. 3 etc.).

The post-processor 32 is configured to perform post-processing on the decoded image data 31 (also referred to as reconstructed image data) such as a decoded image to obtain post-processed image data 33 such as a post-processed image. Post-processing performed by post-processing unit 32 may include, for example, color format conversion (eg, from YCbCr to RGB), toning, trimming, or resampling, or any other processing used to generate decoded image data 31 for display by display device 34, etc. .

A display device 34 is used to receive post-processed image data 33 to display the image to a user or viewer or the like. Display device 34 may be or include any type of display for representing the reconstructed image, eg, an integrated or external display screen or display. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display ), digital light processor (DLP), or any other type of display.

The coding system 10 also includes a training engine 25 for training the encoder 20 (in particular the intra prediction unit 254 in the encoder 20 ) or the decoder 30 (in particular the intra prediction unit 344 in the decoder 30 ) ) to process the first data block or the second data block obtained by splicing or concatenating the surrounding image blocks of the current image block to generate a probability vector of the current image block.

Although FIG. 1A shows source device 12 and destination device 14 as separate devices, device embodiments may include both source device 12 and destination device 14 or the functions of both source device 12 and destination device 14, ie, include both source device 12 and destination device 14. Device 12 or corresponding function and destination device 14 or corresponding function. In these embodiments, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and/or software or by separate hardware and/or software, or any combination thereof.

As will be apparent to the skilled person from the description, the existence and (exact) division of different units or functions in the source device 12 and/or destination device 14 shown in FIG. 1A may vary depending on the actual device and application .

Encoder 20 (eg, video encoder 20 ) or decoder 30 (eg, video decoder 30 ) or both may be implemented by processing circuitry as shown in FIG. 1B , eg, one or more microprocessors, digital signal processors (digital signal processor, DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), discrete logic, hardware, special-purpose processor for video encoding, or any combination thereof . Encoder 20 may be implemented by processing circuitry 46 to include the various modules discussed with reference to encoder 20 of FIG. 2 and/or any other encoder system or subsystem described herein. Decoder 30 may be implemented by processing circuitry 46 to include the various modules discussed with reference to decoder 30 of FIG. 3 and/or any other decoder system or subsystem described herein. The processing circuitry 46 may be used to perform various operations discussed below. As shown in FIG. 5, if parts of the techniques are implemented in software, a device may store the instructions of the software in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby Implement the techniques of the present invention. One of the video encoder 20 and the video decoder 30 may be integrated in a single device as part of a combined codec (encoder/decoder, CODEC), as shown in FIG. 1B .

Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or stationary device, such as a laptop or laptop, cell phone, smartphone, tablet or tablet, camera, Desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (eg, content service servers or content distribution servers), broadcast receiving equipment, broadcast transmitting equipment, etc., and may not Use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

In some cases, the video coding system 10 shown in FIG. 1A is merely exemplary, and the techniques provided herein may be applicable to video coding settings (eg, video encoding or video decoding) that do not necessarily include the encoding device and the Decode any data communication between devices. In other examples, data is retrieved from local storage, sent over a network, and so on. The video encoding device may encode and store the data in memory, and/or the video decoding device may retrieve and decode the data from the memory. In some examples, encoding and decoding are performed by devices that do not communicate with each other but merely encode data to and/or retrieve and decode data from memory.

1B is an illustrative diagram of an example of a video coding system 40 including video encoder 20 of FIG. 2 and/or video decoder 30 of FIG. 3, according to an exemplary embodiment. Video coding system 40 may include imaging device 41, video encoder 20, video decoder 30 (and/or video encoder/decoder implemented by processing circuit 46), antenna 42, one or more processors 43, a or multiple memory stores 44 and/or display devices 45 .

As shown in Figure IB, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and/or display device 45 can communicate with each other. In different examples, video coding system 40 may include only video encoder 20 or only video decoder 30 .

In some examples, antenna 42 may be used to transmit or receive an encoded bitstream of video data. Additionally, in some instances, display device 45 may be used to present video data. Processing circuitry 46 may include application-specific integrated circuit (ASIC) logic, graphics processors, general purpose processors, and the like. Video coding system 40 may also include an optional processor 43, which may similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Additionally, the memory memory 44 may be any type of memory, such as volatile memory (eg, static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory volatile memory (eg, flash memory, etc.), etc. In a non-limiting example, memory storage 44 may be implemented by cache memory. In other examples, processing circuitry 46 may include memory (eg, cache memory, etc.) for implementing image buffers, and the like.

In some examples, video encoder 20 implemented by logic circuitry may include an image buffer (eg, implemented by processing circuitry 46 or memory memory 44 ) and a graphics processing unit (eg, implemented by processing circuitry 46 ). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuitry 46 to implement the various modules discussed with reference to FIG. 2 and/or any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

In some examples, video decoder 30 may be implemented by processing circuitry 46 in a similar manner to implement various of the types discussed with reference to video decoder 30 of FIG. 3 and/or any other decoder systems or subsystems described herein. module. In some examples, logic circuit-implemented video decoder 30 may include an image buffer (implemented by processing circuit 46 or memory memory 44) and a graphics processing unit (eg, implemented by processing circuit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video decoder 30 implemented by processing circuitry 46 to implement the various modules discussed with reference to Figure 3 and/or any other decoder systems or subsystems described herein.

In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data, indicators, index values, mode selection data, etc., as discussed herein related to encoded video frames, such as data related to encoded partitions (eg, transform coefficients or quantized transform coefficients). , (as discussed) optional indicators, and/or data defining the encoding split). Video coding system 40 may also include video decoder 30 coupled to antenna 42 for decoding the encoded bitstream. Display device 45 is used to present video frames.

It should be understood that for the examples described with reference to the video encoder 20 in the embodiments of the present application, the video decoder 30 may be used to perform the opposite process. With regard to signaling syntax elements, video decoder 30 may be operable to receive and parse such syntax elements, decoding the associated video data accordingly. In some examples, video encoder 20 may entropy encode the syntax elements into an encoded video bitstream. In such instances, video decoder 30 may parse such syntax elements and decode related video data accordingly.

For ease of description, refer to the Versatile Video Coding (VVC) reference software or the ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Motion Picture Experts Group (MPEG) The High-Efficiency Video Coding (HEVC) developed by the Joint Collaboration Team on Video Coding (JCT-VC) describes the embodiments of the present invention. Those of ordinary skill in the art understand that the embodiments of the present invention are not limited to HEVC or VVC.

Encoders and Encoding Methods

FIG. 2 is a schematic block diagram of an example of a video encoder 20 for implementing the techniques of this application. In the example of FIG. 2 , the video encoder 20 includes an input terminal (or input interface) 201 , a residual calculation unit 204 , a transform processing unit 206 , a quantization unit 208 , an inverse quantization unit 210 , an inverse transform processing unit 212 , and a reconstruction unit 214 , a loop filter 220 , a decoded picture buffer (DPB) 230 , a mode selection unit 260 , an entropy encoding unit 270 and an output terminal (or output interface) 272 . Mode selection unit 260 may include inter prediction unit 244 , intra prediction unit 254 , and partition unit 262 . Inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in FIG. 2 may also be referred to as a hybrid video encoder or a hybrid video codec-based video encoder.

Referring to FIG. 2 , the intra-frame prediction module includes a trained target model (also called a neural network), which is used to process the first data block or the second data obtained by splicing or concatenating the surrounding image blocks of the current image block or the second data block. block to generate a probability vector for the current image block.

The residual calculation unit 204, the transform processing unit 206, the quantization unit 208 and the mode selection unit 260 constitute the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop The path filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 constitute the backward signal path of the encoder, wherein the backward signal path of the encoder 20 corresponds to the decoding signal path of the decoder (see decoder 30 in Figure 3). Inverse quantization unit 210 , inverse transform processing unit 212 , reconstruction unit 214 , loop filter 220 , decoded image buffer 230 , inter prediction unit 244 , and intra prediction unit 254 also make up the “built-in decoder” of video encoder 20 .

Image and image segmentation (images and blocks)

The encoder 20 may be operable to receive images (or image data) 17, eg, images in a sequence of images forming a video or video sequence, via an input 201 or the like. The received image or image data may also be a preprocessed image (or preprocessed image data) 19 . For simplicity, the following description uses image 17. The image 17 may also be referred to as the current image or the image to be encoded (especially when distinguishing the current image from other images in video encoding, such as the same video sequence, i.e. the video sequence that also includes the current image, previously encoded in the post image and/or post decoded image).

A (digital) image is or can be viewed as a two-dimensional array or matrix of pixel points with intensity values. The pixels in the array may also be called pixels or pels (short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the size and/or resolution of the image. In order to represent color, three color components are usually used, that is, an image can be represented as or include an array of three pixel points. In RBG format or color space, an image includes an array of corresponding red, green and blue pixel points. However, in video coding, each pixel is usually represented in a luma/chroma format or color space, such as YCbCr, including a luma component denoted by Y (and sometimes L) and two chroma components denoted by Cb and Cr. The luminance (luma) component Y represents the luminance or gray level intensity (eg, both are the same in a grayscale image), while the two chrominance (chroma) components Cb and Cr represent the chrominance or color information components . Correspondingly, an image in YCbCr format includes a luminance pixel array of luminance pixel value (Y) and two chrominance pixel arrays of chrominance values (Cb and Cr). Images in RGB format can be converted or transformed to YCbCr format and vice versa, the process is also known as color transformation or conversion. If the image is black and white, the image may only include an array of luminance pixels. Correspondingly, the image may be, for example, a luminance pixel array in monochrome format or a luminance pixel array and two corresponding chrominance pixel arrays in 4:2:0, 4:2:2 and 4:4:4 color formats .

In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit (not shown in FIG. 2 ) for segmenting the image 17 into a plurality of (generally non-overlapping) image blocks 203 . These blocks may also be referred to as root blocks, macroblocks (H.264/AVC) or Coding Tree Blocks (CTBs), or Coding Tree Units (CTUs) in the H.265/HEVC and VVC standards ). The segmentation unit can be used to use the same block size for all images in a video sequence and use a corresponding grid that defines the block size, or to vary the block size between images or subsets of images or groups of images, and to segment each image into corresponding piece.

In other embodiments, the video encoder may be used to directly receive blocks 203 of the image 17 , eg, one, several or all of the blocks that make up the image 17 . The image block 203 may also be referred to as a current image block or an image block to be encoded.

Like image 17 , image block 203 is also or can be considered as a two-dimensional array or matrix of pixels with intensity values (pixel values), but image block 203 is smaller than image 17 . In other words, block 203 may include an array of pixels (eg, a luminance array in the case of a monochrome image 17 or a luminance array or a chrominance array in the case of a color image) or three arrays of pixels (eg, in the case of a color image 17 ) (one luma array and two chrominance arrays) or any other number and/or type of arrays depending on the color format employed. The number of pixels in the horizontal and vertical directions (or axes) of the block 203 defines the size of the block 203 . Correspondingly, the block may be an array of M×N (M columns×N rows) pixel points, or an array of M×N transform coefficients, or the like.

In one embodiment, the video encoder 20 shown in FIG. 2 is used to encode the image 17 block by block, eg, performing encoding and prediction on each block 203 .

In one embodiment, the video encoder 20 shown in FIG. 2 may also be used to segment and/or encode an image using slices (also referred to as video slices), where an image may use one or more slices (typically non-overlapping slices) ) for segmentation or encoding. Each slice may include one or more blocks (eg, Coding Tree Unit CTUs) or one or more groups of blocks (eg, coding tiles in the H.265/HEVC/VVC standard and tiles in the VVC standard ( brick).

In one embodiment, the video encoder 20 shown in FIG. 2 may also be used to use slice/coding block groups (also referred to as video coding block groups) and/or coding blocks (also referred to as video coding blocks) ) to segment and/or encode an image, wherein the image may be segmented or encoded using one or more slices/encoded block groups (usually non-overlapping), each slice/encoded block group may include one or more slices/encoded block groups A block (eg, CTU) or one or more coding blocks, etc., wherein each coding block may be rectangular or the like, and may include one or more full or partial blocks (eg, CTUs).

residual calculation

The residual calculation unit 204 is configured to calculate the residual block 205 according to the image block 203 and the prediction block 265 (the prediction block 265 will be described in detail later) in the following manner: The pixel value of the prediction block 265 is subtracted from the value to obtain the residual block 205 in the pixel domain.

transform

The transform processing unit 206 is configured to perform discrete cosine transform (discrete cosine transform, DCT) or discrete sine transform (discrete sine transform, DST) etc. on the pixel point values of the residual block 205 to obtain transform coefficients 207 in the transform domain. Transform coefficients 207, which may also be referred to as transform residual coefficients, represent the residual block 205 in the transform domain.

Transform processing unit 206 may be used to apply integer approximations of DCT/DST, such as transforms specified for H.265/HEVC. Compared to the orthogonal DCT transform, this integer approximation is usually scaled by some factor. In order to maintain the norm of the forward and inversely transformed residual blocks, other scaling factors are used as part of the transformation process. The scaling factor is usually chosen according to certain constraints, such as the scaling factor being a power of 2 for the shift operation, the bit depth of the transform coefficients, the trade-off between accuracy and implementation cost, etc. For example, specific scaling factors are specified for the inverse transform by the inverse transform processing unit 212 at the encoder 20 side (and for the corresponding inverse transform at the decoder 30 side by, for example, the inverse transform processing unit 312), and accordingly, can be used at the encoder The 20 side specifies the corresponding scaling factor for the forward transformation through the transformation processing unit 206 .

In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206 ) may be configured to output transform parameters such as the type of one or more transforms, eg, directly or after being encoded or compressed by the entropy encoding unit 270 , eg, so that video decoder 30 can receive and decode using transform parameters.

quantify

The quantization unit 208 is configured to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization, to obtain quantized transform coefficients 209 . The quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209 .

The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207 . For example, n-bit transform coefficients may be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, with scalar quantization, different degrees of scaling can be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, while larger quantization step sizes correspond to coarser quantization. A suitable quantization step size can be indicated by a quantization parameter (QP). For example, the quantization parameter may be an index into a predefined set of suitable quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (smaller quantization step size), a larger quantization parameter may correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may include dividing by the quantization step size, and corresponding or inverse dequantization performed by the inverse quantization unit 210 or the like may include multiplying by the quantization step size. Embodiments according to some standards such as HEVC may be used to use quantization parameters to determine the quantization step size. In general, the quantization step size can be calculated from the quantization parameter using a fixed-point approximation of an equation involving division. Other scaling factors may be introduced for quantization and dequantization to restore the norm of the residual block, which may be modified due to the scale used in the fixed-point approximation of the equations for the quantization step size and quantization parameters. In one exemplary implementation, the inverse transform and dequantized scales may be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream etc. Quantization is a lossy operation, where the larger the quantization step size, the larger the loss.

In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) may be used to output a quantization parameter (QP), eg, directly or after being encoded or compressed by the entropy encoding unit 270, eg, such that the video Decoder 30 may receive and decode using the quantization parameters.

inverse quantization

The inverse quantization unit 210 is used to perform inverse quantization of the quantization unit 208 on the quantized coefficients to obtain the dequantized coefficients 211, for example, perform inverse quantization with the quantization scheme performed by the quantization unit 208 according to or using the same quantization step size as the quantization unit 208 Program. Dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, corresponding to transform coefficients 207, but due to losses caused by quantization, inverse quantized coefficients 211 are usually not identical to transform coefficients.

Inverse transform

The inverse transform processing unit 212 is used to perform the inverse transform of the transform performed by the transform processing unit 206, for example, an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to A reconstructed residual block 213 (or corresponding dequantized coefficients 213) is obtained. The reconstructed residual block 213 may also be referred to as a transform block 213 .

reconstruction

The reconstruction unit 214 (eg, summer 214 ) is used to add the transform block 213 (ie, the reconstructed residual block 213 ) to the prediction block 265 to obtain the reconstructed block 215 in the pixel domain, eg, the The pixel value and the pixel value of the prediction block 265 are added.

filter

The loop filter unit 220 (or "loop filter" 220 for short) is used to filter the reconstruction block 215 to obtain the filter block 221, or generally to filter the reconstructed pixels to obtain filtered pixel values. For example, loop filter units are used to smooth pixel transitions or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as self- Adaptive loop filter (ALF), noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter. The order of the filtering process can be deblocking filter, SAO filter and ALF filter. As another example, a process called luma mapping with chroma scaling (LMCS) (ie, adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. )edge. Although loop filter unit 220 is shown in FIG. 2 as a loop filter, in other configurations, loop filter unit 220 may be implemented as a post-loop filter. Filter block 221 may also be referred to as filter reconstruction block 221 .

In one embodiment, video encoder 20 (correspondingly, loop filter unit 220) may be used to output loop filter parameters (eg, SAO filter parameters, ALF filter parameters, or LMCS parameters), eg, directly or by entropy The encoding unit 270 performs entropy encoding and outputs, eg, so that the decoder 30 can receive and decode using the same or different loop filter parameters.

decoded image buffer

A decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use by the video encoder 20 in encoding the video data. DPB 230 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), Resistive RAM (RRAM) or other types of storage devices. A decoded image buffer 230 may be used to store one or more filter blocks 221. The decoded image buffer 230 may also be used to store other previously filtered blocks of the same current image or a different image, such as a previous reconstructed image, such as the previously reconstructed and filtered block 221, and may provide a complete previously reconstructed or decoded image (and corresponding reference blocks and pixels) and/or a partially reconstructed current image (and corresponding reference blocks and pixels), eg for inter prediction. The decoded image buffer 230 may also be used to store one or more unfiltered reconstructed blocks 215, or generally unfiltered reconstructed pixels, eg, reconstructed blocks 215 not filtered by the in-loop filtering unit 220, or unfiltered Any other processed reconstructed blocks or reconstructed pixels.

Mode selection (segmentation and prediction)

Mode selection unit 260 includes partition unit 262, inter prediction unit 244, and intra prediction unit 254 for receiving or obtaining original blocks from decoded image buffer 230 or other buffers (eg, column buffers, not shown) 203 (current block 203 of current image 17) and original image data such as reconstructed image data, e.g. filtered and/or unfiltered reconstructed pixels or reconstructions of the same (current) image and/or one or more previously decoded images piece. The reconstructed image data is used as reference image data required for prediction such as inter prediction or intra prediction to obtain the prediction block 265 or the prediction value 265 .

The mode selection unit 260 may be used to determine or select a partition for the current block prediction mode (including no partition) and prediction mode (eg, intra or inter prediction mode) to generate a corresponding prediction block 265 for performing the residual block 205 The reconstruction block 215 is computed and reconstructed.

In one embodiment, mode selection unit 260 may be used to select a partitioning and prediction mode (eg, from among those supported or available by mode selection unit 260) that provides the best match or the smallest residual (minimum Residual refers to better compression in transmission or storage), or provides minimal signaling overhead (minimum signaling overhead refers to better compression in transmission or storage), or considers or balances both. The mode selection unit 260 may be configured to determine the segmentation and prediction mode according to rate distortion optimization (RDO), ie select the prediction mode that provides the least rate distortion optimization. The terms "best", "lowest", "optimal", etc. herein do not necessarily refer to "best", "lowest", "optimal" in general, but may also refer to situations where termination or selection criteria are met, for example, Values above or below the threshold or other constraints may result in "sub-optimal choices" but reduce complexity and processing time.

In other words, partition unit 262 may be used to partition pictures in a video sequence into a sequence of coding tree units (CTUs), CTU 203 may be further partitioned into smaller block parts or sub-blocks (blocks again), e.g., Quad-tree partitioning (QT) partitioning, binary-tree partitioning (BT) partitioning, or triple-tree partitioning (TT) partitioning or any combination thereof is used by iteration, and for e.g. Or each of the sub-blocks performs prediction, wherein the mode selection includes selecting the tree structure of the partition block 203 and selecting a prediction mode to apply to each of the block parts or sub-blocks.

The segmentation (eg, performed by segmentation unit 262 ) and prediction processing (eg, performed by inter-prediction unit 244 and intra-prediction unit 254 ) performed by video encoder 20 will be described in detail below.

segmentation

Partition unit 262 may partition (or divide) one coding tree unit 203 into smaller parts, such as square or rectangular shaped pieces. For an image with three pixel arrays, a CTU consists of N×N luminance pixel blocks and two corresponding chrominance pixel blocks. The maximum allowable size of a luma block in a CTU is specified as 128x128 in the under-development Versatile Video Coding (VVC) standard, but may be specified to a value other than 128x128 in the future, such as 256x256. The CTUs of a picture can be aggregated/grouped into slices/coded block groups, coded blocks or bricks. A coding block covers a rectangular area of an image, and a coding block can be divided into one or more tiles. A brick consists of multiple CTU lines within an encoded block. An encoded block that is not divided into multiple bricks can be called a brick. However, bricks are a true subset of coded blocks and are therefore not called coded blocks. VVC supports two encoding block group modes, namely raster scan slice/encoded block group mode and rectangular slice mode. In raster scan coded block group mode, a slice/coded block group contains a sequence of coded blocks in a raster scan of coded blocks of an image. In rectangular slice mode, slices contain multiple tiles of an image that together make up a rectangular area of the image. The tiles within the rectangular slice are arranged in the order of the tile raster scan of the photo. These smaller blocks (also referred to as sub-blocks) may be further divided into smaller parts. This is also known as tree splitting or hierarchical tree splitting, where a root block at root tree level 0 (hierarchy level 0, depth 0) etc. can be recursively split into two or more blocks of the next lower tree level, For example, a node at tree level 1 (hierarchy level 1, depth 1). These blocks can in turn be split into two or more blocks of the next lower level, e.g. tree level 2 (hierarchy level 2, depth 2), etc., until the split ends (since ending criteria are met, such as reaching a maximum tree depth or minimum block size). Blocks that are not further divided are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary-tree (BT), a tree divided into three parts is called a ternary-tree (TT), and a tree divided into four parts is called a quadtree ( quad-tree, QT).

For example, a coding tree unit (CTU) may be or include a CTB for luma pixels, two corresponding CTBs for chroma pixels for an image with an array of three pixels, or a CTB for pixels for monochrome images, or a CTB using three The CTB of a pixel of an image encoded by the independent color plane and syntax structure (used to encode the pixel). Correspondingly, a coding tree block (CTB) can be a block of N×N pixel points, where N can be set to a certain value such that the components are divided into CTBs, which is division. A coding unit (CU) may be or include a coding block of luminance pixels, two corresponding coding blocks of chrominance pixels of an image with an array of three pixel points, or a coding block of pixels of a monochrome image, or An encoding block of pixels of an image encoded using three independent color planes and syntax structures (used to encode pixels). Correspondingly, a coding block (CB) can be a block of M×N pixel points, where M and N can be set to a certain value so that the CTB is divided into coding blocks, which is division.

For example, in an embodiment, a coding tree unit (CTU) may be divided into multiple CUs according to HEVC by using a quad-tree structure represented as a coding tree. The decision whether to use inter (temporal) prediction or intra (spatial) prediction to encode image regions is made at the leaf-CU level. Each leaf-CU may be further divided into one, two, or four PUs according to the PU partition type. The same prediction process is used within a PU, and relevant information is transmitted to the decoder on a PU basis. After applying the prediction process to obtain residual blocks according to the PU partition type, the leaf CU may be partitioned into transform units (TUs) according to other quad-tree structures similar to the coding tree used for the CU.

For example, in an embodiment, according to the latest video coding standard currently under development, called Versatile Video Coding (VVC), a combined quadtree of nested multi-type trees (eg, binary and ternary trees) is used to partition for segmentation coding The segmented structure of the tree unit.In the coding tree structure in the coding tree unit, the CU can be a square or a rectangle.For example, the coding tree unit (CTU) is first divided by the quad-tree structure.The quad-leaf node is further composed of multiple types of Tree structure division. There are four division types for multi-type tree structures: vertical binary tree division (SPLIT_BT_VER), horizontal binary tree division (SPLIT_BT_HOR), vertical ternary tree division (SPLIT_TT_VER) and horizontal ternary tree division (SPLIT_TT_HOR). Multi-type leaf nodes are called A coding unit (CU), unless the CU is too large for the maximum transform length, such a segment is used for prediction and transform processing without any other partitioning. In most cases, this means that the CU, PU, and TU are The block size is the same in the coding block structure of tree-nested multi-type trees. This exception occurs when the maximum supported transform length is less than the width or height of the color components of the CU. VVC has formulated a multi-type tree with quadtree nesting The only signaling mechanism for partitioning information in the coding structure. In the signaling mechanism, the coding tree unit (CTU) as the root of the quad-tree is first divided by the quad-tree structure. Then each quad-leaf node (when enough can be further divided into a multi-type tree structure. In the multi-type tree structure, whether the node is further divided by the first flag (mtt_split_cu_flag), when the node is further divided, first use the second flag (mtt_split_cu_vertical_flag) to indicate Divide the direction, and then use the third mark (mtt_split_cu_binary_flag) to indicate that the division is binary tree division or ternary tree division. According to the values of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the decoder can derive the multi-type tree division mode (MttSplitMode) of the CU based on a predefined rule or table. It should be noted that for a certain design, such as a 64×64 luma block and a 32×32 chroma pipeline design in a VVC hardware decoder, when the width or height of the luma coding block is greater than 64, TT division is not allowed .When the width or height of the chroma coding block is greater than 32, TT division is also not allowed. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), and each VPDU is defined in the image as mutual Non-overlapping units. In hardware decoders, successive VPDUs are processed simultaneously in multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is necessary to keep the VPD small U. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the partition of ternary tree (TT) and binary tree (BT) may increase the size of VPDU.

In addition, it should be noted that when a part of the tree node block exceeds the bottom or the right border of the image, the tree node block is forced to be divided until all the pixels of each coded CU are located within the image border.

For example, the intra sub-partitions (ISP) tool may divide the luma intra prediction block vertically or horizontally into two or four sub-parts depending on the block size.

In one example, mode selection unit 260 of video encoder 20 may be used to perform any combination of the partitioning techniques described above.

As described above, video encoder 20 is used to determine or select the best or optimal prediction mode from a set of (predetermined) prediction modes. The set of prediction modes may include, for example, intra prediction modes and/or inter prediction modes.

Intra prediction

The set of intra prediction modes may include 35 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined by HEVC, or may include 67 different Intra prediction modes, for example, non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC. For example, several conventional-angle intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes for non-square blocks defined in VVC. As another example, in order to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of the non-square block. In addition, the intra prediction result of the planar mode may also be modified using a position-dependent intra prediction combination (PDPC) method.

The intra-frame prediction unit 254 is configured to generate an intra-frame prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to the intra-frame prediction mode in the intra-frame prediction mode set.

The intra prediction unit 254 (or usually the mode selection unit 260) is further configured to output intra prediction parameters (in this embodiment of the present application, the intra prediction parameters include the information of the target intra prediction mode of the current image block and the current image block). The probability vector of ) is sent to entropy encoding unit 270 in syntax element 266 for inclusion into encoded image data 21 so that video decoder 30 may perform operations such as receiving and using prediction parameters for decoding.

Inter prediction

In a possible implementation, the set of inter-prediction modes depends on available reference pictures (i.e., for example the aforementioned at least partially previously decoded pictures stored in DBP 230) and other inter-prediction parameters, for example on whether to use the entire reference picture or Use only a part of the reference image, e.g. the search window area near the area of the current block, to search for the best matching reference block, and/or e.g. depending on whether half pixel, quarter pixel and/or within 16th of a pixel is performed Interpolated pixel interpolation.

In addition to the above prediction modes, skip mode and/or direct mode may also be employed.

For example, extending merge prediction, the merge candidate list for this mode consists of the following five candidate types in order: spatial MVP from spatially adjacent CUs, temporal MVP from collocated CUs, history-based MVP from FIFO table, For average MVP and zero MV. Decoder side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of MV for merged mode. The merge mode with MVD (MMVD) comes from the merge mode with motion vector difference. Send the MMVD flag immediately after sending the skip flag and merge flag to specify whether the CU uses MMVD mode. A CU-level adaptive motion vector resolution (AMVR) scheme may be used. AMVR supports the MVD of the CU to be encoded in different precisions. According to the prediction mode of the current CU, the MVD of the current CU is adaptively selected. When a CU is encoded in combined mode, a combined inter/intra prediction (CIIP) mode may be applied to the current CU. A weighted average is performed on the inter and intra prediction signals to obtain the CIIP prediction. For affine motion compensation prediction, the affine motion field of the block is described by motion information of 2 control points (4 parameters) or 3 control points (6 parameters) motion vectors. Subblock-based temporal motion vector prediction (SbTMVP) is similar to temporal motion vector prediction (TMVP) in HEVC, but predicts the motion of sub-CUs in the current CU vector. Bi-directional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces computation, especially in terms of the number of multiplications and the size of the multipliers. In the triangular division mode, the CU is evenly divided into two triangular parts in two divisions: diagonal division and anti-diagonal division. In addition, the bidirectional prediction mode is extended on the basis of simple averaging to support weighted average of two prediction signals.

Inter prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both not shown in FIG. 2 ). The motion estimation unit may be used to receive or obtain the image block 203 (the current image block 203 of the current image 17 ) and the decoded image 231 , or at least one or more previously reconstructed blocks, eg, one or more other/different previously decoded images 231 . Reconstruction blocks for motion estimation. For example, the video sequence may include the current image and the previous decoded image 231, or in other words, the current image and the previous decoded image 231 may be part of or form a sequence of images forming the video sequence.

For example, the encoder 20 may be operable to select a reference block from a plurality of reference blocks of the same or different pictures among a plurality of other pictures, and convert the reference picture (or reference picture index) and/or the position (x, y coordinates) of the reference block ) and the position of the current block (spatial offset) are provided to the motion estimation unit as inter prediction parameters. This offset is also called a motion vector (MV).

The motion compensation unit is used to obtain, eg, receive, inter-prediction parameters, and perform inter-prediction based on or using the inter-prediction parameters, resulting in the inter-prediction block 246 . The motion compensation performed by the motion compensation unit may involve extracting or generating prediction blocks from motion/block vectors determined through motion estimation, and may also include performing interpolation to sub-pixel precision. Interpolative filtering can generate pixels of other pixels from pixels of known pixels, thereby potentially increasing the number of candidate prediction blocks that can be used to encode an image block. Once the motion vector corresponding to the PU of the current image block is received, the motion compensation unit may locate the prediction block pointed to by the motion vector in one of the reference image lists.

The motion compensation unit may also generate block- and video slice-related syntax elements for use by video decoder 30 in decoding image blocks of the video slice. In addition, or instead of slices and corresponding syntax elements, coding block groups and/or coding blocks and corresponding syntax elements may be generated or used.

Entropy coding

The entropy coding unit 270 is used for entropy coding algorithm or scheme (for example, variable length coding (variable length coding, VLC) scheme, context adaptive VLC scheme (context adaptive VLC, CALVC), arithmetic coding scheme, binarization algorithm, Context adaptive binary arithmetic coding (context adaptive binary arithmetic coding, CABAC), syntax-based context adaptive binary arithmetic coding (syntax-based context-adaptive binary arithmetic coding, SBAC), probability interval partitioning entropy (probability interval partitioning entropy, PIPE) ) coding or other entropy coding method or technique) is applied to the quantized residual coefficients 209, inter prediction parameters, intra prediction parameters, loop filter parameters and/or other syntax elements, resulting in an encoded bit stream that can be passed through output 272 The encoded image data 21 output in the form of 21 or the like, so that the video decoder 30 or the like can receive and use the parameters for decoding. The encoded bitstream 21 may be transmitted to the video decoder 30, or stored in memory for later transmission or retrieval by the video decoder 30.

In this embodiment of the present application, the intra prediction parameters include a target intra prediction mode and a probability vector of the current image block, and the entropy encoding unit 270 selects a first reference value or a second reference value corresponding to the target intra prediction mode from the probability vector. and then encode the first reference value or the second reference value into the code stream, where the first reference value or the second reference value is used to indicate the target intra prediction mode of the current image block.

Other structural variations of video encoder 20 may be used to encode the video stream. For example, the non-transform based encoder 20 may directly quantize the residual signal without transform processing unit 206 for certain blocks or frames. In another implementation, encoder 20 may have quantization unit 208 and inverse quantization unit 210 combined into a single unit.

Decoders and Decoding Methods

FIG. 3 illustrates an example video decoder 30 for implementing the techniques of this application. The video decoder 30 is adapted to receive the encoded image data 21 (eg, the encoded bitstream 21 ) encoded by the encoder 20 , for example, to obtain a decoded image 331 . The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks of an encoded video slice (and/or encoded block groups or encoded blocks) and associated syntax elements.

In the example of FIG. 3, decoder 30 includes entropy decoding unit 304, inverse quantization unit 310, inverse transform processing unit 312, reconstruction unit 314 (eg, summer 314), loop filter 320, decoded image buffer (DBP) ) 330 , a mode application unit 360 , an inter prediction unit 344 and an intra prediction unit 354 . Inter prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform a decoding process that is substantially the inverse of the encoding process described with reference to video encoder 100 of FIG. 2 .

Referring to FIG. 3 , the intra-frame prediction module includes a trained target model (also called a neural network), which is used to process the first data block or the second data obtained by splicing or concatenating the surrounding image blocks of the current image block or the second data block. block to generate a probability vector for the current image block.

As described in the encoder 20, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded image buffer DPB 230, the inter prediction unit 344 and the intra prediction unit 354 also constitute a video encoder 20 "built-in decoders". Accordingly, the inverse quantization unit 310 may be functionally the same as the inverse quantization unit 110, the inverse transform processing unit 312 may be functionally the same as the inverse transform processing unit 122, the reconstruction unit 314 may be functionally the same as the reconstruction unit 214, and the loop Filter 320 may be functionally identical to loop filter 220 , and decoded image buffer 330 may be functionally identical to decoded image buffer 230 . Therefore, the explanations of the corresponding units and functions of the video encoder 20 apply correspondingly to the corresponding units and functions of the video decoder 30 .

Entropy decoding

The entropy decoding unit 304 is used to parse the bit stream 21 (or generally the encoded image data 21 ) and perform entropy decoding on the encoded image data 21 to obtain quantization coefficients 309 and/or decoded encoding parameters (not shown in FIG. 3 ), etc. , such as in inter prediction parameters (such as reference picture indices and motion vectors), intra prediction parameters (such as intra prediction mode or index), transform parameters, quantization parameters, loop filter parameters and/or other syntax elements, etc. any or all. The entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to the encoding scheme of the entropy encoding unit 270 of the encoder 20 . Entropy decoding unit 304 may also be used to provide inter-prediction parameters, intra-prediction parameters, and/or other syntax elements to mode application unit 360 , as well as other parameters to other units of decoder 30 . Video decoder 30 may receive syntax elements at the video slice and/or video block level. In addition, or instead of slices and corresponding syntax elements, encoded block groups and/or encoded blocks and corresponding syntax elements may be received or used.

In this embodiment of the present application, the intra prediction parameters that the entropy decoding unit 304 may be configured to provide to the mode application unit 360 include a target reference value for indicating a target intra prediction mode of the current image block for performing intra prediction.

inverse quantization

Inverse quantization unit 310 may be operable to receive quantization parameters (QPs) (or information related to inverse quantization in general) and quantization coefficients from encoded image data 21 (eg, parsed and/or decoded by entropy decoding unit 304), and based on The quantization parameters inverse quantize the decoded quantized coefficients 309 to obtain inverse quantized coefficients 311 , which may also be referred to as transform coefficients 311 . The inverse quantization process may include using quantization parameters calculated by video encoder 20 for each video block in the video slice to determine the degree of quantization, as well as the degree of inverse quantization that needs to be performed.

Inverse transform

An inverse transform processing unit 312 may be operable to receive dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain a reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 313 . The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. Inverse transform processing unit 312 may also be operable to receive transform parameters or corresponding information from encoded image data 21 (eg, parsed and/or decoded by entropy decoding unit 304 ) to determine transforms to apply to dequantized coefficients 311 .

reconstruction

The reconstruction unit 314 (eg, summer 314) is used to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the pixel domain, for example, the pixel point values of the reconstructed residual block 313 and the prediction block 365 pixel values are added.

filter

The loop filter unit 320 (in or after the encoding loop) is used to filter the reconstruction block 315 to obtain a filter block 321, so as to smoothly perform pixel transitions or improve video quality, etc. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a self- Adaptive loop filter (ALF), noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter. The order of the filtering process can be deblocking filter, SAO filter and ALF filter. As another example, a process called luma mapping with chroma scaling (LMCS) (ie, adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. )edge. Although loop filter unit 320 is shown in FIG. 3 as a loop filter, in other configurations, loop filter unit 320 may be implemented as a post-loop filter.

decoded image buffer

The decoded video block 321 in one picture is then stored in a decoded picture buffer 330 which stores the decoded picture 331 as a reference picture for subsequent motion compensation of other pictures and/or output display respectively.

The decoder 30 is configured to output the decoded image 311 through the output terminal 312, etc., to display to the user or for the user to view.

predict

The inter prediction unit 354 may be functionally the same as the inter prediction unit 244 (in particular, the motion compensation unit), the intra prediction unit 344 may be functionally the same as the intra prediction unit 254, and is based on the encoded image data 21 (e.g. The received partitioning and/or prediction parameters or corresponding information are parsed and/or decoded by the entropy decoding unit 304 to decide the partitioning or partitioning and perform prediction. In the present application, the intra prediction unit 344 may determine the target intra prediction mode of the current image block for intra prediction based on the target reference value obtained in the entropy decoding unit 204 and the probability vector obtained in the neural network model. The mode application unit 360 may be configured to perform prediction (intra or inter prediction) of each block according to the reconstructed image, block or corresponding pixel points (filtered or unfiltered), resulting in a prediction block 365 .

When encoding a video slice as an intra-coded (I) slice, the intra-prediction unit 354 in the mode application unit 360 is used to generate data based on the indicated intra-prediction mode and data from previously decoded blocks of the current image. Prediction block 365 for an image block of the current video slice. When a video image is encoded as an inter-coded (ie, B or P) slice, an inter-prediction unit 344 (eg, a motion compensation unit) in the mode application unit 360 is used to decode the motion vector and other syntax received from the entropy decoding unit 304 according to the motion vector The element generates a prediction block 365 for a video block of the current video slice. For inter prediction, these prediction blocks may be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct reference frame List 0 and List 1 from reference pictures stored in DPB 330 using default construction techniques. In addition to or instead of slices (eg, video slices), the same or similar process may be applied to embodiments of coding block groups (eg, video coding block groups) and/or coding blocks (eg, video coding blocks), For example, video may be encoded using I, P, or B encoding block groups and/or encoding blocks.

Mode application unit 360 is operable to determine prediction information for a video block of the current video slice by parsing motion vectors and other syntax elements, and use the prediction information to generate a prediction block for the current video block being decoded. For example, mode applying unit 360 uses some of the received syntax elements to determine a prediction mode (eg, intra-prediction or inter-prediction), an inter-prediction slice type (eg, B-slice, P-slice, or GPB for encoding a video block of the video slice) slice), construction information for one or more reference picture lists of the slice, motion vectors for each inter-coded video block of the slice, inter-prediction status for each inter-coded video block of the slice, other information to decode video blocks within the current video slice. In addition to or instead of slices (eg, video slices), the same or similar process may be applied to embodiments of coding block groups (eg, video coding block groups) and/or coding blocks (eg, video coding blocks), For example, video may be encoded using I, P, or B encoding block groups and/or encoding blocks.

In one embodiment, the video encoder 30 shown in FIG. 3 may also be used to segment and/or decode images using slices (also referred to as video slices), where an image may use one or more slices (typically non-overlapping slices) ) for segmentation or decoding. Each slice may include one or more blocks (eg, CTUs) or one or more groups of blocks (eg, coded blocks in the H.265/HEVC/VVC standard and bricks in the VVC standard.

In one embodiment, the video decoder 30 shown in FIG. 3 may also be used to use slice/coding block groups (also referred to as video coding block groups) and/or coding blocks (also referred to as video coding blocks) ) to segment and/or decode an image, wherein the image may be segmented or decoded using one or more slices/encoded block groups (usually non-overlapping), each slice/encoded block group may include one or more A block (eg, CTU) or one or more coding blocks, etc., wherein each coding block may be rectangular or the like, and may include one or more full or partial blocks (eg, CTUs).

Other variations of the video decoder 30 may be used to decode the encoded image data 21 . For example, decoder 30 may generate the output video stream without loop filter unit 320 . For example, the non-transform based decoder 30 may directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, video decoder 30 may have inverse quantization unit 310 and inverse transform processing unit 312 combined into a single unit.

It should be understood that in the encoder 20 and the decoder 30, the processing result of the current step can be further processed, and then output to the next step. For example, after interpolation filtering, motion vector derivation or loop filtering, further operations, such as clip or shift operations, may be performed on the processing results of interpolation filtering, motion vector derivation or loop filtering.

It should be noted that further operations may be performed on the derived motion vectors of the current block (including but not limited to control point motion vectors in affine mode, affine, plane, sub-block motion vectors in ATMVP mode, temporal motion vectors, etc.). For example, the value of the motion vector is limited to a predefined range according to the representation bits of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents a power. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (eg, the MVs of four 4x4 subblocks in an 8x8 block) is limited such that the maximum difference between the integer parts of the four 4x4 subblock MVs does not More than N pixels, eg no more than 1 pixel. There are two ways to limit motion vectors based on bitDepth.

Although the above embodiments have primarily described video codecs, it should be noted that embodiments of the coding system 10, encoder 20 and decoder 30, as well as other embodiments described herein, may also be used for still image processing or codecs, That is, the processing or coding of a single image in video codecs that is independent of any previous or consecutive images. In general, if image processing is limited to a single image 17, inter prediction unit 244 (encoder) and inter prediction unit 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of video encoder 20 and video decoder 30 are also available for still image processing, such as residual calculation 204/304, transform 206, quantization 208, inverse quantization 210/310, (inverse ) transform 212/312, partition 262/362, intra prediction 254/354 and/or loop filtering 220/320, entropy encoding 270 and entropy decoding 304.

FIG. 4 is a schematic diagram of a video decoding apparatus 400 according to an embodiment of the present invention. Video coding apparatus 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video coding apparatus 400 may be a decoder, such as the video decoder 30 in FIG. 1A, or an encoder, such as the video encoder 20 in FIG. 1A.

The video decoding apparatus 400 includes: an input port 410 (or input port 410) for receiving data and a receiver unit (receiver unit, Rx) 420; a processor, a logic unit or a central processing unit (central processing unit) for processing data , CPU) 430; for example, the processor 430 here can be a neural network processor 430; a transmitter unit (transmitter unit, Tx) 440 for transmitting data and an output port 450 (or output port 450); memory 460. The video coding apparatus 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 410, the receiving unit 420, the transmitting unit 440, and the output port 450, Exit or entrance for optical or electrical signals.

The processor 430 is implemented by hardware and software. Processor 430 may be implemented as one or more processor chips, cores (eg, multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the ingress port 410 , the receiving unit 420 , the sending unit 440 , the egress port 450 and the memory 460 . The processor 430 includes a decoding module 470 (eg, a neural network NN based decoding module 470). The decoding module 470 implements the embodiments disclosed above. For example, the transcoding module 470 performs, processes, prepares or provides various encoding operations. Thus, a substantial improvement in the functionality of the video coding apparatus 400 is provided by the coding module 470, and switching of the video coding apparatus 400 to different states is affected. Alternatively, decoding module 470 is implemented as instructions stored in memory 460 and executed by processor 430 .

Memory 460 includes one or more magnetic disks, tape drives, and solid-state drives, and may serve as an overflow data storage device for storing programs when such programs are selected for execution, and for storing instructions and data read during program execution. Memory 460 may be volatile and/or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content addressable memory (ternary) content-addressable memory, TCAM) and/or static random-access memory (SRAM).

FIG. 5 is a simplified block diagram of an apparatus 500 provided by an exemplary embodiment, and the apparatus 500 can be used as either or both of the source device 12 and the destination device 14 in FIG. 1A .

The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices, existing or to be developed in the future, capable of manipulating or processing information. Although the disclosed implementations may be implemented using a single processor, such as processor 502 as shown, using more than one processor is faster and more efficient.

In one implementation, the memory 504 in the apparatus 500 may be a read only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 504 . Memory 504 may include code and data 506 accessed by processor 502 via bus 512 . The memory 504 may also include an operating system 508 and application programs 510 including at least one program that allows the processor 502 to perform the methods described herein. For example, applications 510 may include applications 1 through N, and also include video coding applications that perform the methods described herein.

Apparatus 500 may also include one or more output devices, such as display 518 . In one example, display 518 may be a touch-sensitive display that combines a display with touch-sensitive elements that may be used to sense touch input. Display 518 may be coupled to processor 502 through bus 512 .

Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. In addition, secondary storage may be directly coupled to other components of the device 500 or accessed through a network, and may include a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Accordingly, the apparatus 500 may have various configurations.

Since the embodiments of the present application involve the application of neural networks, for ease of understanding, some nouns or terms used in the embodiments of the present application are explained below, and the nouns or terms are also part of the content of the invention.

(1) Neural network

Neural Network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs and intercept 1 as inputs. The output of the operation unit can be:

Among them, s=1, 2,...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be an area composed of several neural units.

(2) Deep neural network

Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the division of DNN according to the position of different layers, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the middle layers are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, in terms of the work of each layer, it is not complicated. In short, it is the following linear relationship expression:

in,

is the input vector,

is the output vector,

is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is just an input vector

After such a simple operation to get the output vector

Due to the large number of DNN layers, the coefficient W and the offset vector

The number is also much larger. These parameters are defined in the DNN as follows: Take the coefficient W as an example: Suppose that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as

The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. The summary is: the coefficient from the kth neuron in the L-1 layer to the jth neuron in the Lth layer is defined as

It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better capture the complexities of the real world. In theory, a model with more parameters is more complex and has a larger "capacity", which means that it can complete more complex learning tasks. Training the deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by the vectors W of many layers).

(3) Convolutional Neural Network

Convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. Learning at multiple levels. As a deep learning architecture, CNN is a feed-forward artificial neural network in which individual neurons can respond to images fed into it. A convolutional neural network consists of a feature extractor consisting of convolutional and pooling layers. The feature extractor can be viewed as a filter, and the convolution process can be viewed as convolution with an input image or a convolutional feature map using a trainable filter.

The convolutional layer refers to the neuron layer in the convolutional neural network that convolves the input signal. The convolution layer can include many convolution operators. The convolution operator is also called the kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator can essentially is a weight matrix, which is usually pre-defined, during the convolution operation on the image, the weight matrix is usually one pixel by one pixel (or two pixels by two pixels) along the horizontal direction on the input image... ...it depends on the value of stride) to process, so as to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix will be extended to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will result in a single depth dimension of the convolutional output, but in most cases a single weight matrix is not used, but multiple weight matrices of the same size (row × column) are applied, That is, multiple isotype matrices. The output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" described above. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to extract unwanted noise in the image. Blur, etc. The multiple weight matrices have the same size (row×column), and the size of the feature maps extracted from the multiple weight matrices with the same size is also the same, and then the multiple extracted feature maps with the same size are combined to form a convolution operation. output. The weight values in these weight matrices need to be obtained through a lot of training in practical applications, and each weight matrix formed by the weight values obtained by training can be used to extract information from the input image, so that the convolutional neural network can make correct predictions. When the convolutional neural network has multiple convolutional layers, the initial convolutional layer often extracts more general features, which can also be called low-level features; as the depth of the convolutional neural network deepens, The features extracted by the later convolutional layers are more and more complex, such as features such as high-level semantics, and the features with higher semantics are more suitable for the problem to be solved.

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer, which can be a convolutional layer followed by a pooling layer, or a multi-layer convolutional layer followed by a layer or multiple pooling layers. During image processing, the only purpose of pooling layers is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a max pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the pixel values in the image within a certain range to produce an average value as the result of average pooling. The max pooling operator can take the pixel with the largest value in the range as the result of max pooling. Also, just as the size of the weight matrix used in the convolutional layer should be related to the size of the image, the operators in the pooling layer should also be related to the size of the image. The size of the output image after processing by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer.

After processing by the convolutional layer/pooling layer, the convolutional neural network is not enough to output the required output information. Because as mentioned before, convolutional/pooling layers will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (required class information or other relevant information), the convolutional neural network needs to utilize neural network layers to generate one or a set of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers may be obtained by pre-training according to the relevant training data of a specific task type. For example, the task type may include image recognition, Image classification, image super-resolution reconstruction, and more.

Optionally, after the multi-layer hidden layers in the neural network layer, it also includes the output layer of the entire convolutional neural network, which has a loss function similar to categorical cross-entropy, specifically for calculating the prediction error, once the entire volume The forward propagation of the convolutional neural network is completed, and the backpropagation will start to update the weight values and biases of the aforementioned layers to reduce the loss of the convolutional neural network, and the result and ideal output of the convolutional neural network through the output layer. error between results.

(4) Loss function

In the process of training a deep neural network, because it is hoped that the output of the deep neural network is as close as possible to the value you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then based on the difference between the two to update the weight vector of each layer of neural network (of course, there is usually an initialization process before the first update, that is, to pre-configure parameters for each layer in the deep neural network), for example, if the predicted value of the network If it is high, adjust the weight vector to make its prediction lower, and keep adjusting until the deep neural network can predict the real desired target value or a value that is very close to the real desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function (loss function) or objective function (objective function), which are used to measure the difference between the predicted value and the target value. important equation. Among them, taking the loss function as an example, the higher the output value of the loss function (loss), the greater the difference, then the training of the deep neural network becomes the process of reducing the loss as much as possible.

The target model (also referred to as a neural network) for intra prediction will be described in detail below with reference to FIG. 6 . FIG. 6 illustrates an example architecture 600 of a target model, such as a neural network for prediction, or prediction network for short. Taking the first data block or the second data block as the input of the neural network, the neural network uses the convolution layer to process the input data, and uses the Softmax layer to output the probability vector of the current image block.

The main structure of the neural network model is the deep residual network (ResNet) in the convolutional neural network. ResNet consists of 9 residual blocks (residual blocks); each residual block has two basic layers, Each base layer consists of a convolution layer (convolution layer), a non-linear activation function (rectified linear unit, ReLU) layer, and a batch normalization (BN) layer. In addition, there is a convolutional layer between the first residual block and the data input, which can speed up the convergence speed of neural network model training and alleviate the influence of gradient disappearance or gradient explosion phenomenon on training. After the last residual block, a convolutional layer and an adaptive average pooling layer are connected in turn to convert the feature map into a multi-dimensional vector, and finally 67 intra-frame prediction modes are output through the Softmax layer. The probability distribution of , which is a multi-dimensional vector whose dimension is the sum of the number of intra-frame prediction modes shown in FIG. 9 , that is, 67.

It should be understood that the neural network model used in the embodiments of the present application is only a specific example, which is not limited in the embodiments of the present application. Those skilled in the art can also construct a neural network model through other architectures, so as to achieve the same technical effect as the neural network model in the embodiments of the present application, for example: a convolutional neural network including AlexNet, ZFNet, VGGNet, GoogleNet, etc. A neural network model consisting of one or more of the architectures or non-convolutional neural network architectures. In addition, those skilled in the art can also achieve the same technical effect as the embodiment of the present application by changing part of the structure of the neural network model in the embodiment of the present application, for example, changing the number of residual blocks of the neural network model in the present application.

After designing the initial neural network model structure for calculating the probability vector, start training the neural network model. The training process of the neural network model in the embodiment of the present application is as follows:

In this embodiment of the present application, the initial neural network model is trained on the image processor. The training database of the neural network model includes a training set, a validation set and a test set, and the training set, the validation set and the test set respectively include a series of image sets. In the following, the first data block or the second data block in the input neural network model is called the target data block. For each size of the target data block, select the target data block from the training set, validation set and test set respectively. Images with the same size are used as the training set, validation set and test set of the target data block, each image in the training set of the target data block is compressed, and the input data and target data of the neural network model are extracted. Each image is a data block obtained after compression, and the target data is the first identifier corresponding to the target intra prediction mode of each image. In the process of each training, the compressed data block is input into the neural network model, and a 67-dimensional probability vector corresponding to the data block is obtained, and the cross entropy loss function shown in formula (1-2) is used to measure the In the second training process, the error between the output of the neural network model and the expected, so as to judge whether to end the training of the neural network model.

Specifically, when the cross-entropy loss function shown in formula (1-2) is used to calculate the value of the cross-entropy loss function after each training, M represents the number of categories of samples, where M is the total number of intra-frame prediction modes of the image, that is, M=67; i represents the target data of the image used in each training, that is, the first identifier corresponding to the target intra prediction mode of the image; T(i) is an indicator function, and when the indicated category is the same as the sample category, it is 1, otherwise its value is 0. For example, when the target data of the current image is 30, when i is 30, the value of T(i) is 1; when i is the rest of the values (0-29 or 31-66) Any value), the value of T(i) is 0; P _i represents the value of the i-th dimension in the probability distribution output by the Softmax layer. It should be understood that the larger the value of _Pi , that is, the larger the probability of using the target intra prediction mode to perform intra prediction, the smaller the loss function value will be.

After each training and the calculation of each cross-entropy loss function value, determine the relationship between each cross-entropy loss function value and the preset loss value, when each cross-entropy loss function value is greater than the preset loss value , adjust the weight of each parameter in the neural network model according to the value of each cross-entropy loss function, and use the adjusted neural network model for the next training, until the cross-entropy loss function value is less than or equal to the preset loss value, end the neural network Training of the network model. Then, use the verification set and test set of the target data block to verify and test the neural network model respectively; after verification and testing, the parameters of the neural network model corresponding to the target data block can be determined, that is, the parameters corresponding to the target data block can be obtained. The neural network model corresponding to the target data block. The above operations are respectively performed on target data blocks of each size, and then neural network models with different parameter configurations corresponding to target data blocks of different sizes can be obtained.

It should be understood that the embodiments of the present application may adopt different training databases according to different application scenarios, which are not specifically limited in the embodiments of the present application. In addition, the training method for the neural network model in the embodiment of the present application is only a specific example, which is not specifically limited in the present application. In addition to the above methods, other methods can also be used to train the neural network model. For example, firstly, the initial neural network model is trained, verified and tested by using the training set, the validation set and the test set containing images of a certain size, and an intermediate model is determined. Neural network model, and then using the method described in the above embodiment, for target data blocks of different sizes, the intermediate neural network model is trained, verified and tested respectively, and different parameters corresponding to target data blocks of different sizes are obtained. The configured neural network model, wherein the above-mentioned specific size may be any size selected according to the actual scene, which is not specifically limited here.

It should be understood that target data blocks of different sizes correspond to neural network models with different parameter configurations. After the above-mentioned size transformation operation, the size of the second data block is less than the size of the first data block. The storage space of the neural network model corresponding to the data block.

The process of encoding and decoding the intra prediction mode of the current image block by using the method proposed in this application will be described in detail below.

FIG. 7 is a flowchart of a method 700 for encoding a target intra prediction mode of a current image block in an embodiment of the present application. The method 700 may be performed by the video encoder 20 , and in particular, may be performed by the intra prediction unit 254 and the entropy encoding unit 270 of the video encoder 20 . The method 700 is described as a series of steps or operations, and it should be understood that the execution order of some steps of the method 700 is not limited, such as steps S701 and S702. The method 700 shown in FIG. 7 includes steps S701, S702, S703 and S704, which will be described in detail below.

Step S701, acquiring at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks of the current image block.

Specifically, three surrounding image blocks on the left, upper left and upper sides adjacent to the current image block are obtained, and the size of the three surrounding image blocks is the same as that of the current image block (as shown in FIG. 10 ); Each surrounding image block includes a reconstruction block, a residual block and a prediction block, and at least two of the reconstruction block, residual block and prediction block of each image block in the three surrounding image blocks and the current image block are selected. In a feasible implementation manner, the reconstruction block, residual block and prediction block of each image block in the three surrounding image blocks and the current image block may be selected for subsequent splicing or concatenation operations.

Step S702, obtain a probability vector of the current image block according to at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks and the prediction mode probability model, and multiple elements in the probability vector are one with multiple intra prediction modes. One-to-one correspondence, any element in the probability vector is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block.

In a feasible implementation manner, when the target intra-frame prediction mode of the current image block is the non-matrix weighted intra-frame prediction MIP mode, according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image block , the current image block and the prediction mode probability model to obtain the probability vector of the current image block.

The above-mentioned obtaining the probability vector of the current image block according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks and the prediction mode probability model, specifically includes: At least two of the blocks are spliced or concatenated to obtain a first data block; and a probability vector of the current image block is obtained according to the first data block and the prediction mode probability model. It should be noted that the sizes of the first data blocks obtained by the two methods of splicing and concatenation are different.

Wherein, by splicing or concatenating at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks, the process of obtaining the first data block can be divided into four parts according to the positional relationship of the current image block in the current frame This is the case: that is, when the current image block is located in four different positions of the upper left edge, left edge, upper edge and non-edge of the current frame (corresponding to the four positions in Figure 11-1 to Figure 11-4) relationship), the three surrounding image blocks of the current image block are selected in different ways. In the embodiments of the present application, the width and height of the current image block are respectively M and N for detailed description, and M and N are positive integers.

The first positional relationship, as shown in Figure 11-1, when the current image block is located at the non-edge position of the current frame, that is, the distances between the current image block and the left edge and the upper edge of the current frame are greater than or equal to M, When N, the three surrounding image blocks on the left, the upper left and the upper side that are equal in size to the current image block and adjacent to the current image block can be directly selected from the current frame. Each image block in the block contains a reconstruction block, residual block and prediction block; for the current image block, there is no data in the residual block, reconstruction block and prediction block of the current image block, and the residual block of the current image block has no data. The difference block, the reconstruction block and the prediction block are filled with a default value or 0, wherein the default value is 2 ⁿ -1, and n is the pixel bit depth inside the video encoder 100 .

Optionally, in the first positional relationship, when the reconstruction block, residual block and prediction block of each image block in the three surrounding image blocks and the current image block are selected, and the three surrounding coded blocks and the current coded block are selected. When splicing, the obtained first data block is a three-channel data block with a size of 2M×2N, and the three channels of the first data block are filled with the reconstruction blocks of the current image block and the surrounding image blocks, respectively. Residual block and prediction block; when selecting the reconstruction block and prediction block of each image block in the three surrounding image blocks and the current image block, or the reconstruction block and the residual block, or the prediction block and the residual block, and for the three When the surrounding coding block and the current coding block are spliced, the obtained first data block is a two-channel data block with a size of 2M×2N, and the two channels of the first data block are filled with the current image block and the current image block respectively. Reconstructed and predicted blocks of surrounding image blocks.

Optionally, in the first positional relationship, if the data filled in the residual block, the reconstruction block and the prediction block of the current image block are all 0, select the three surrounding image blocks and the data of each image block in the current image block. When the reconstruction block, the residual block and the prediction block are concatenated, and the three surrounding coding blocks and the current coding block are concatenated, the obtained first data block is a data block with nine channels and a size of M×N; when selecting three The reconstruction block and prediction block of each image block in the surrounding image block and the current image block, and when the three surrounding coding blocks and the current coding block are concatenated, the first data block obtained is a four-channel and the size is M× N data blocks; it should be understood that when the data blocks selected for cascading in each image block in the surrounding image block and the current image block are different, and the data filled in each data block of the current image block is different, the level The number of channels of the first data block obtained by concatenating is different. Except for the two cases listed above, in other cases, the number of channels of the first data block obtained by concatenating the surrounding image blocks and the current image block is easy to derive. obtained, and will not be repeated here.

The second positional relationship, as shown in Figure 11-2, is when the current image block is located at the upper left edge of the current frame, that is, when the distance between the current image block and the left edge and upper edge of the current frame is smaller than M and N respectively , the three surrounding image blocks on the left, upper left and upper sides that are the same size as the current image block and adjacent to the current image block cannot be selected from the current frame. At this time, the adjacent left, upper left and upper sides of the current image block are respectively expanded out. Three surrounding image blocks with the same size as the current image block; in this positional relationship, the reconstruction block and prediction block in each image block of the three surrounding image blocks are filled with the default value 2 ⁿ -1, each The residual block in the image block is filled with 0 or 2 ⁿ -1; the reconstruction block, residual block and prediction block of the current image block can be filled with 0 or 2 ⁿ -1.

Optionally, in the second positional relationship, when the reconstruction block and the prediction block in each of the three surrounding image blocks are filled with a default value of 2 ⁿ -1, the residual block in each image block is Fill in 0, and fill in 0 into the reconstruction block, residual block and prediction block of the current image block; when selecting the reconstruction block, residual block and prediction block of each image block in the three surrounding image blocks and the current image block , and when splicing the three surrounding coding blocks and the current coding block, the obtained first data block is a two-channel data block with a size of 2M×2N; when selecting each of the three surrounding image blocks and the current image block When the residual block and prediction block of the image block are spliced together with the three surrounding coding blocks and the current coding block, the obtained first data block is a one-channel data block with a size of 2M × 2N; it should be understood that when the surrounding The image block and each image block in the current image block have different data blocks selected for splicing, and the current image block and the data filled in each of the three surrounding image blocks are different, the first image block obtained by splicing is different. The number of channels of the data blocks is different. Except for the two cases listed above, in other cases, the number of channels of the first data block obtained by splicing the surrounding image blocks and the current image block can be easily derived. Repeat.

Optionally, in the second positional relationship, when the reconstruction block and the prediction block in each of the three surrounding image blocks are filled with a default value of 2 ⁿ -1, the residual block in each image block is Fill in 0, and fill in 0 into the reconstruction block, residual block and prediction block of the current image block; when selecting the reconstruction block, residual block and prediction block of each image block in the three surrounding image blocks and the current image block , and when the three surrounding coding blocks and the current coding block are concatenated, the obtained first data block is a data block with four channels and a size of M×N; it should be understood that when the surrounding image blocks and the current image block are When the data blocks selected for concatenation in each image block are different, and the data filled in each data block in the current image block and the surrounding image blocks are different, the number of channels of the first data block obtained by concatenation is different. Except for the cases listed above, in other cases, the number of channels of the first data block obtained by concatenating the surrounding image blocks and the current image block can be easily derived, which will not be repeated here.

The third positional relationship, as shown in Figure 11-3, is when the current image block is located at the left edge of the current frame, that is, the distance between the current image block and the left edge of the current frame is less than M, and the distance between the current image block and the current frame is less than M. When the distance of the upper edge is greater than or equal to N, at this time, an upper image block adjacent to the current image block and with the same size can be selected from the current frame, and expanded to the left and upper left adjacent to the current image block respectively. two surrounding image blocks with the same size as the current image block; in this positional relationship, the reconstruction block and prediction block in each image block of the extended left and upper left image blocks are filled with default The value is 2 ⁿ -1, the residual block in each image block is filled with 0 or 2 ⁿ -1; the reconstruction block, residual block and prediction block of the current image block can be filled with 0 or 2 ⁿ -1.

Optionally, in the third positional relationship, when a default value of 2 ⁿ -1 is filled in the reconstruction block and the prediction block in each image block of the left and upper left image blocks, the residual value in each image block is The difference block is filled with 0, and 0 is filled in the reconstruction block, residual block and prediction block of the current image block; when three surrounding image blocks and the reconstruction block, residual block and When predicting the block and splicing the three surrounding coding blocks and the current coding block, the obtained first data block is a two-channel data block with a size of 2M×2N; when selecting the three surrounding image blocks and the current image block When splicing the residual block and prediction block of each image block, and splicing the three surrounding coding blocks and the current coding block, the obtained first data block is a one-channel data block with a size of 2M×2N; it should be understood that, When the data blocks selected for splicing are different between the surrounding image block and each image block in the current image block, and the data filled in each data block of the current image block and the three surrounding image blocks are different, the splicing result The number of channels of the first data block is different. Except for the two cases listed above, in other cases, the number of channels of the first data block obtained by splicing the surrounding image block and the current image block can be easily derived. Here No longer.

Optionally, in the third positional relationship, when the reconstruction block, prediction block and residual block in each image block of the left and upper left image blocks are filled with a default value of 2 ⁿ -1, the current image block is sent to the current image block. 0 is filled in the reconstructed block and residual block of , and 2 ⁿ -1 is filled in the prediction block; when three surrounding image blocks and the reconstruction block, residual block and prediction block of each image block in the current image block are selected, When the three surrounding coding blocks and the current coding block are concatenated, the obtained first data block is a data block with ten channels and a size of M×N; it should be understood that when each of the surrounding image blocks and the current image block is When the data blocks selected for concatenation are different from the image blocks, and the data filled in each data block in the current image block and the surrounding image blocks are different, the number of channels of the first data block obtained by concatenation is different, except In addition to the cases listed above, in other cases, the number of channels of the first data block obtained by concatenating the surrounding image blocks and the current image block can be easily derived, and details are not repeated here.

The fourth positional relationship, as shown in Figure 11-4, is when the current image block is located at the left edge of the current frame, that is, the distance between the current image block and the left edge of the current frame is greater than or equal to M, and the current image block is located at the left edge of the current frame. When the distance between the upper edge of the frame is less than N, at this time, a left image block of the same size adjacent to the current image block can be selected from the current frame, and then expanded to the adjacent upper and upper left sides of the current image block. Two surrounding image blocks with the same size as the current image block are obtained. In this positional relationship, the reconstructed block and prediction block in each image block of the extended upper and upper left image blocks are filled with default values. The value is 2 ⁿ -1, the residual block in each image block can be filled with 0 or 2 ⁿ -1; the reconstruction block, residual block and prediction block of the current image block can be filled with 0 or 2 ⁿ -1.

Optionally, in the fourth positional relationship, when the reconstruction block and the prediction block in each image block of the upper and upper left image blocks are filled with a default value of 2 ⁿ -1, the residual error in each image block is The block is filled with 0, and 0 is filled in the reconstruction block, residual block and prediction block of the current image block; when three surrounding image blocks and the reconstruction block, residual block and prediction block of each image block in the current image block are selected When the three surrounding coding blocks and the current coding block are spliced, the first data block obtained is a two-channel data block with a size of 2M×2N; when three surrounding image blocks and each of the current image blocks are selected When splicing the residual block and prediction block of two image blocks, and splicing the three surrounding coding blocks and the current coding block, the obtained first data block is a one-channel data block with a size of 2M×2N; it should be understood that when When the data blocks selected for splicing are different between the surrounding image block and each image block in the current image block, and the data filled in each data block of the current image block and the three surrounding image blocks are different, the splicing obtained The number of channels of a data block is different. Except for the two cases listed above, in other cases, the number of channels of the first data block obtained by splicing the surrounding image block and the current image block can be easily derived. Repeat.

Optionally, in the fourth positional relationship, when the reconstruction block, prediction block and residual block in each image block of the upper and upper left image blocks are filled with a default value of 2 ⁿ -1, the current image block is sent to the current image block. 0 is filled in the reconstructed block and residual block of , and 2 ⁿ -1 is filled in the prediction block; when three surrounding image blocks and the reconstruction block, residual block and prediction block of each image block in the current image block are selected, When the three surrounding coding blocks and the current coding block are concatenated, the obtained first data block is a data block with ten channels and a size of M×N; it should be understood that when each of the surrounding image blocks and the current image block is When the data blocks selected for concatenation are different from the image blocks, and the data filled in each data block in the current image block and the surrounding image blocks are different, the number of channels of the first data block obtained by concatenation is different, except In addition to the cases listed above, in other cases, the number of channels of the first data block obtained by concatenating the surrounding image blocks and the current image block can be easily derived, and details are not repeated here.

In addition, the image block described in this application may be understood as, but not limited to, a prediction unit (prediction unit, PU), a coding unit CU, or a transformation unit TU, and so on. According to the provisions of different video compression codec standards, a CU may include one or more prediction unit PUs. For example, in HEVC, one CU may include multiple PUs, but in VVC, one CU corresponds to one PU. Image blocks may have fixed or variable sizes, and vary in size according to different video compression codec standards. In addition, the current image block refers to an image block to be encoded or decoded currently, such as a prediction unit to be encoded or decoded.

In a feasible implementation manner, obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model may include: when the size of the first data block is the target size, inputting the first data block into the prediction The mode probability model is processed to obtain the probability vector of the current image block, or; when the size of the first data block is not the target size, a size transformation operation is performed on the first data block to obtain the second data block, the second data block The size of the block is equal to the target size, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or proportional scaling in both horizontal and vertical directions; input the second data block into The prediction mode probability model is processed to obtain the probability vector of the current image block.

The above-mentioned probability mode prediction model may be the neural network model shown in FIG. 6 , or other forms of neural network models, or mathematical models, which are not specifically limited in this application.

Table 1: Size categories of the first data block

Wherein, for current image blocks of different sizes, after splicing or concatenating the current image block and three surrounding coding blocks of the current image block, the sizes of the first data blocks obtained are different. In the embodiment of the present application, there are 17 types of sizes of the first data block (as shown in Table 1), and a size transformation operation is performed on the first data block to obtain a second data block, and the size of the second data block is less than 17 types. kind.

When the size of the first data block is not the target size, the process of performing a size transformation operation on the first data block to obtain the second data block can be divided into four ways:

(1) When the size of the first data block is not the target size, and the ratio of the first side length to the second side length of the first data block is equal to the ratio of the first side length to the second side length of the target size, then The first data block is proportionally scaled in both horizontal and vertical directions to obtain a second data block;

(2) When the size of the first data block is not the target size, and the ratio of the first side length to the second side length of the first data block is equal to the ratio of the second side length to the first side length of the target size, correct the Transpose the first data block, and proportionally scale the horizontal and vertical directions to obtain the second data block;

(3) When the size of the first data block is not the target size, and the first side length and the second side length of the first data block are respectively equal to the second side length and the first side length of the target size, The block is transposed to obtain the second data block;

(4) When the size of the first data block is not the target size, scaling the first data block to obtain a second data block.

Optionally, there can be seven types of target sizes, that is, the ratio of the first side length to the second side length from Table 1 is 1:1, 1:2, 2:1, 1:4, 4:1, One of the sizes of 1:8 and 8:1 is selected as the target size. Compare the size of the first data block with the seven target sizes. According to the relationship between the size of the first data block and the seven target sizes, one of the above four methods (1), (2) or (3) can be used. In any of the ways, the second data block is obtained.

For example, the process of obtaining the second data block in the above embodiment is described in detail by taking the length of the first side and the length of the second side of the first data block as 16×8: if the length of the first side and the length of the second side in the target size are The ratio is 2:1, and the first side length and the second side length of the target size are 32×16 respectively, then the first side length and the second side length of the first data block are measured in horizontal and vertical directions respectively. Proportional enlargement to obtain a second data block with the first side length and the second side length respectively being 32×16; if the ratio of the first side length and the second side length in the target size is 2:1, the The side length and the second side length are respectively 8×4, then the first side length and the second side length of the first data block are scaled down in the horizontal and vertical directions respectively, and the first side length and the second side length are obtained. The second data block whose two sides are 8×4 respectively. It should be understood that the specific process of scaling mentioned in other parts of the embodiments of the present application is the same as here.

Optionally, there may be four types of target sizes, that is, one is selected from the sizes in Table 1 where the ratio of the length of the first side to the length of the second side is 1:1, and the size of 1:2 or 2:1 Choose one of the sizes of 1:4 or 4:1, and choose one of the sizes of 1:8 or 8:1. There are four sizes in total as the target size. Compare the size of the first data block with the four target sizes. According to the relationship between the size of the first data block and the four target sizes, (1) or (2) or (3) of the above four methods can be used. In any of the ways, the second data block is obtained.

It should be noted that, in the embodiments of the present application, when transposing and scaling processing of the first data block is required, the transposing and scaling operations are performed in no order, and the The two data blocks have the same size.

It can be seen that in the embodiment of the present application, through the above two implementations, the size types of the first data block can be reduced from 17 to 7 or 4 types of the second data block, thereby effectively reducing the subsequent The type of the neural network model corresponding to the second data block of the size, thereby saving the storage space for storing the neural network model and improving the coding performance.

Optionally, the target size may be one of the 17 sizes in Table 1, or one other than the 17 sizes in Table 1. The size of the first data block is compared with the one target size, and according to the relationship between the size of the first data block and the four target sizes, the above-mentioned method (4) can be used to obtain the second data block.

For example, the first side length and the second side length of the target size are 4×8 to describe the specific process implemented in the above embodiment in detail: when the first side length and the second side length of the first data block size are 4 When ×4, the second side length of the first data block is enlarged in the vertical direction to obtain a second data block whose first side length and second side length are 4×8 respectively; when the size of the first data block is When the first side length and the second side length are respectively 16×4, the first side length and the second side length of the first data block are reduced in the horizontal direction and enlarged in the vertical direction, respectively, to obtain the sum of the first side length and the second side length. The second data blocks whose second side lengths are respectively 4×8.

It can be seen that, in this embodiment of the present application, the size types of the first data block can be reduced from 17 types to 1 type, and the size types of the second data block are reduced to the greatest extent, so that one type of data block that is related to the second data block can be obtained subsequently. The neural network model corresponding to the block size significantly reduces the storage space for storing the neural network model and improves the coding efficiency; at the same time, the first data block with simple texture and large size is reduced to a smaller size After the second data block, the computational complexity of the subsequent calculation of the probability vector using the neural network model can be effectively reduced, thereby improving the coding performance.

In a feasible implementation manner, obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model may include: when the size of the first data block is the target size, inputting the first data block into the prediction The pattern probability model is processed to obtain the probability vector of the current image block, or;

Specifically, the first data block is first determined according to the size relationship between the first side length and the second side length of the first data block, and the size relationship between the absolute value of the horizontal gradient and the sum of the absolute value of the vertical gradient and the preset threshold. The corresponding target size; when the size of the first data block is not the target size, a size transformation operation is performed on the first data block to obtain the second data block.

Wherein, optionally, the value of the preset threshold may be 4*M*N, the length of the first side and the length of the second side of the current image block are M and N respectively, the length of the first side and the length of the first side of the first data block are respectively M and N. The lengths of the two sides are 2M and 2N respectively; the horizontal gradient of the current image block and the vertical gradient of the current image block are calculated by the Sobel operator. It should be understood that the value of the preset threshold can be determined according to the actual situation, which is not limited in this embodiment of the present application; in addition, the operator for calculating the horizontal gradient and the vertical gradient is not limited to the Sobel operator, and can be determined according to the actual situation, for example , you can also choose Scharr operator, laplacian operator or other operators with similar functions.

The gradient of the image represents the change of the image gray level, and the horizontal gradient and vertical gradient of the current image block respectively represent the gray level change of the current image block in the horizontal and vertical directions relative to the surrounding image blocks in the current frame. The image can be understood as a two-dimensional discrete function f(x,y), the gradient of the image f(x,y) at point (x,y) is a vector with size and direction, let Gx and Gy represent the current image block respectively The gradient in the horizontal and vertical directions, the vector of this gradient can be expressed as:

The magnitude of this vector is:

The direction angle of this vector is:

When the size of the first data block is not the target size, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, the first data block The process of performing the size transformation operation on the block to obtain the second data block can be divided into two cases:

(1) The above-mentioned size transformation operation includes scaling, and the size of the first data block is scaled according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block. The transformation operation to obtain the second data block includes: when the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the absolute value of the horizontal gradient and the absolute value of the vertical gradient are summed When the sum is smaller than the preset threshold, the first data block is scaled to obtain the second data block, and the first side length and the second side length of the second data block are 4M/N, 4 respectively; When the first side length M is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to a preset threshold, the first data block is scaled to The second data block is obtained, and the first side length and the second side length of the second data block are 8M/N, 8 respectively; when the first side length M of the first data block is less than the second side length N of the first data block , and when the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset threshold, the first data block is scaled to obtain the second data block, the first side length and the second side length of the second data block are respectively 4, 4N/M; when the first side length M of the first data block is less than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the preset When the threshold is set, the first data block is scaled to obtain a second data block, and the first side length and the second side length of the second data block are respectively 8, 8N/M; wherein, M and N are positive integers.

(2) The above-mentioned size transformation operation includes at least one of scaling and transposition, the above-mentioned size relationship according to the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block. , performing a size transformation operation on the first data block to obtain a second data block, including:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset threshold, the first data block Scaling is performed to obtain a second data block, and the first side length and the second side length of the second data block are 4M/N, 4 respectively; when the first side length M of the first data block is greater than or equal to the first data block When the second side length is N, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the preset threshold, the first data block is scaled to obtain the second data block, the The length of one side and the length of the second side are 8M/N, 8 respectively;

Step S703, determining the target intra prediction mode of the current image block.

Optionally, search out the 67 intra-frame prediction modes (as shown in The target intra-frame prediction mode for intra-frame prediction of the image block, the mode value (0-66) of the target intra-frame prediction mode in FIG. 9 is the corresponding first identifier.

9, 67 intra-frame prediction modes in VVC include direct current (direct current, DC) prediction mode, plane Planar prediction mode and 65 angle prediction modes, and the first identifiers of DC and Planar prediction modes are 1 and 1 respectively. 0, the first identifiers of the remaining 65 angle prediction modes are 2-66 from the lower left corner to the upper right corner, respectively.

Step S704, according to the target intra-frame prediction mode and the probability vector, encode the target intra-frame prediction mode into the code stream.

According to the target intra-frame prediction mode and the probability vector, encoding the target intra-frame prediction mode into the code stream can be divided into two cases:

(1) When the size transformation operation does not include transposition, determine the first reference value according to the first identifier and the probability vector of the target intra-frame prediction mode, and encode the first reference value to obtain the corresponding target intra-frame prediction mode. code stream.

Among them, the mode value of the target intra prediction mode in FIG. 9 is the corresponding first identifier, the probability vector is a 67-dimensional vector, and each element in the probability vector corresponds to a probability interval on the interval (0, 1). , and the elements in the probability vector correspond one-to-one with the 67 pattern values in Figure 9.

The above-mentioned first identification and probability vector of the target intra-frame prediction mode determine a first reference value, and encode the first reference value to obtain a code stream corresponding to the target intra-frame prediction mode, specifically: according to the target intra-frame prediction mode The first identification of the first identification determines the corresponding element of the first identification in the probability vector, and then arbitrarily selects a value in the probability interval corresponding to the element as the first reference value to be encoded into the code stream, and the first reference value can be used to characterize the current The target intra prediction mode of the image block.

(2) when the size transformation operation includes transposition, and the target intra-frame prediction mode is the angle prediction mode, the second mark is determined according to the preset constant and the first mark of the target intra-frame prediction mode, according to the second mark and the probability vector, Determine the second reference value; encode the second reference value to obtain a code stream corresponding to the target intra-frame prediction mode; when the size transformation operation includes transposition, and the target intra-frame prediction mode is a non-angle prediction mode, according to the target frame The first identifier and probability vector of the intra prediction mode are used to determine the first reference value; the first reference value is encoded to obtain a code stream corresponding to the target intra prediction mode.

Wherein, the preset constant is the number of the multiple intra-frame prediction modes shown in FIG. 3 , that is, 67, the angle prediction mode is the intra-frame prediction mode with mode values 2-66, and the non-angle prediction mode is the mode value of DC DC mode or flat mode.

Specifically, when the size transformation operation includes transposition and the target intra-frame prediction mode is the angle prediction mode, the first identifier is subtracted from the preset constant to obtain the second identifier, and the corresponding identifier of the second identifier is selected from the probability vector. element, and then arbitrarily select a value in the probability interval corresponding to this element as the second reference value and encode it into the code stream; when the size transformation operation includes transposition, and the target intra-frame prediction mode is non-angle prediction mode, the target intra-frame The coding method of the prediction mode is the same as that of the method (1), which is not repeated here.

It can be seen that, in the embodiment shown in FIG. 7 , at least two of the reconstruction blocks, prediction blocks and residual blocks of the three surrounding image blocks of the current image block are obtained, since the reconstruction of each surrounding image block can Two of the block, the prediction block and the residual block are calculated to obtain the other, so the embodiment of the present application can make full use of the relevant information of the surrounding image blocks, and then according to the reconstruction block, the prediction block and the residual block of the surrounding image blocks The probability vectors of at least two generated current image blocks are more accurate, which improves the accuracy of the data encoded in the code stream and improves the encoding performance; The size transformation operation is performed to obtain the second data block, which can effectively reduce the number of size types of the second data block, thereby reducing the number of neural network models corresponding to the second data blocks of different sizes, thereby saving the storage space of the neural network model, Improve coding performance; at the same time, for the first data block with simple texture and large size, after the size transformation operation is reduced to a second data block with a smaller size, it can effectively reduce the subsequent use of the neural network model to calculate the probability vector. computational complexity, thereby improving coding efficiency.

Please refer to FIG. 8 , which is a flowchart of another method 800 for encoding a target intra prediction mode of a current image block in the present application. As shown in Figure 8, the method 800 includes the following six steps:

S801, acquiring surrounding image blocks of the current image block.

Specifically, three surrounding image blocks on the left, upper left and upper sides adjacent to the current image block can be obtained, the size of the three surrounding image blocks is the same as that of the current image block (as shown in FIG. 10 ), and the three surrounding image blocks are Each surrounding image block in the block contains a reconstruction block, a residual block and a prediction block, and one or more of the reconstruction block, residual block and prediction block of each image block in the three surrounding image blocks and the current image block can be selected. for subsequent splicing or cascading operations.

S802, splicing or concatenating surrounding image blocks to obtain a first data block.

S803, perform a size transformation operation on the first data block to obtain a second data block, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or both horizontal and vertical scaling A proportional scaling of the direction.

S804, input the second data block into the prediction mode probability model for processing to obtain a probability vector of the current image block, where multiple elements in the probability vector correspond to multiple intra prediction modes one-to-one, and any element in the probability vector uses Indicates the probability of using the intra prediction mode corresponding to any element when predicting the current image block.

S805: Determine the target intra prediction mode of the current image block.

S806, according to the target intra-frame prediction mode and the probability vector, encode the target intra-frame prediction mode into the code stream.

Specifically, steps S802 to S804 are respectively the same as the processes corresponding to step S702, and steps S805 and S806 are respectively the same as the corresponding steps in steps S703 and S704, which will not be repeated here.

It can be seen that in the embodiment shown in FIG. 8 , there are various sizes of the first data block. By performing a size transformation operation on the first data block to obtain the second data block, the size types of the second data block can be effectively reduced , thereby reducing the number of neural network models corresponding to second data blocks of different sizes, thereby saving the storage space of neural network models and improving coding performance; at the same time, for the first data block with simpler texture and larger size , after the size transformation operation is reduced to a second data block with a smaller size, the computational complexity of the subsequent calculation of the probability vector by using the neural network model can be effectively reduced, thereby improving the coding efficiency.

Please refer to FIG. 12. FIG. 12 is a flowchart of an encoder encoding related syntax elements of an intra prediction mode of a current image block according to an embodiment of the present application. As shown in FIG. 12, in VVC, for a current image block, the syntax elements related to the intra prediction mode include the MIP flag bit, the multi-reference line MRL index, the flag corresponding to the MIP mode and the ISP mode, the The first identifier corresponding to the target intra prediction mode of the current image block, and the probability vector of the current image block; when the entropy encoder 103 encodes the syntax elements related to the intra prediction of the current image block, it is divided into the following steps: The description in step S704, select the first reference value or the second reference value corresponding to the target intra-frame prediction mode from the probability vector; use the conventional encoding method to encode the MIP flag bit, and determine whether the MIP flag bit is the first preset value, if it is, then adopt the bypass encoding method to encode the MIP mode; if the MIP flag bit is the second preset value, then adopt the conventional encoding method to encode the MRL index, the corresponding sign of the ISP mode in turn, and adopt the bypass encoding method to encode the current The first reference value or the second reference value for intra prediction of the image block using the target intra prediction mode, wherein the first preset value and the second preset value are 1 and 0 respectively. It should be noted that the above-mentioned first preset value and second preset value are only a specific example in the embodiment of the present application, which is not specifically limited in the present application.

Among them, the information such as the MIP flag bit, the MRL index and the ISP mode have been determined in the process of searching the target intra prediction mode of the current image block, and the MIP flag bit is used to indicate whether the current image block adopts the MIP mode for intra prediction. The MRL index indicates the index of the reference line used for intra prediction of the current image block, and the ISP mode indicates the sub-block division method of the current image block, wherein all sub-blocks of the current image block share one intra prediction mode, and in the current image block When the image block adopts MRL, ISP is not performed.

It should be understood that the encoder 20 described in the embodiments of the present application uses the context-adaptive binary arithmetic coding CABAC method to encode the syntax elements related to the intra prediction of the current image block. In this application, the current image can also be encoded in other ways The syntax elements related to intra-block prediction are encoded, which is not specifically limited here.

Please refer to FIG. 13 . FIG. 13 is a flowchart of a method 1300 for decoding a target intra prediction mode of a current image block in an embodiment of the present application. The method 1300 shown in FIG. 13 includes steps S1301 , S1302 and S1303 , these steps are described in detail below.

Step S1301: Acquire a code stream corresponding to the current image block and at least two of the reconstruction blocks, prediction blocks and residual blocks of surrounding image blocks of the current image block.

Specifically, the decoder 30 obtains the video code stream encoded by the entropy encoder 304; the specific process of obtaining at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks of the current image block is the same as that of step S701. It is not repeated here.

Step S1302: Obtain a probability vector of the current image block according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks and the prediction mode probability model, and multiple elements in the probability vector are one with multiple intra prediction modes. One-to-one correspondence, any element in the probability vector is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block.

Specifically, the specific execution process of step S1302 is the same as that of step S702, which is not repeated here.

Step S1303: Determine the target intra prediction mode of the current image block according to the code stream and the probability vector, and determine the prediction block of the current image block according to the target intra prediction mode.

Specifically, the code stream is decoded to obtain the target reference value; the target intra prediction mode of the current image block is determined according to the target reference value.

The above-mentioned determination of the target intra prediction mode of the current image block according to the target reference value includes two situations:

(1) When the size transformation operation does not include transposition, the target intra prediction mode is determined according to the target reference value and the probability vector.

Specifically, the target reference value is between 0 and 1, the corresponding identifier in the probability vector is determined according to the probability interval in which the target reference value falls, and the target intra prediction mode of the current image block is determined according to the corresponding identifier.

(2) when the size transformation operation includes transposition, determine the first identifier according to the target reference value and the probability vector; when the intra prediction mode corresponding to the first identifier is the angle prediction mode, determine according to the preset constant and the first identifier target intra-frame prediction mode; when the intra-frame prediction mode corresponding to the first identifier is a non-angular prediction mode, the target intra-frame prediction mode is determined according to the target reference value and the probability vector.

Specifically, the first identifier of the element corresponding to the target reference value in the probability vector is determined according to the probability interval in which the target reference value falls, and when the intra prediction mode corresponding to the first identifier is the angle prediction mode, a preset constant is used to subtract the The first mark obtains the second mark, and the intra-frame prediction mode corresponding to the second mark in FIG. 9 is the target intra-frame prediction mode of the current image block; when the intra-frame prediction mode corresponding to the first mark is not In the case of the angle prediction mode, the intra prediction mode corresponding to the first identifier in FIG. 9 is the target intra prediction mode of the current image block.

Please refer to FIG. 14 . FIG. 14 is a flowchart of a method 1400 for decoding a target intra prediction mode of a current image block in an embodiment of the present application. The method 1400 shown in FIG. 14 includes steps S1401 , S1402 and 1403 , S1404 and S1405, these steps will be described in detail below.

S1401: Acquire a code stream corresponding to the current image block and surrounding image blocks of the current image block.

Specifically, acquiring the code stream corresponding to the current image block above is the same as the corresponding step in step S1301, and acquiring the surrounding image blocks of the current image block above is the same as the corresponding step in step S801, which is not repeated here.

S1402, splicing or concatenating surrounding image blocks to obtain a first data block.

S1403, perform a size transformation operation on the first data block to obtain a second data block, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or both horizontal and vertical scaling A proportional scaling of the direction.

S1404: Input the second data block into the prediction mode probability model for processing to obtain a probability vector of the current image block. Multiple elements in the probability vector correspond to multiple intra prediction modes one-to-one, and any element in the probability vector uses Indicates the probability of using the intra prediction mode corresponding to any element when predicting the current image block.

S1405: Determine the target intra prediction mode of the current image block according to the code stream and the probability vector, and determine the prediction block of the current image block according to the target intra prediction mode.

Specifically, the specific execution processes of steps S1402 , S1403 , S1404 and S1405 are the same as those of S802 , S803 , S804 and S1303 respectively, and will not be repeated here.

Please refer to FIG. 15 , which is a schematic block diagram of an encoding apparatus 1500 provided by the present application. It should be understood that the encoding apparatus 1500 here may correspond to the encoder 20 in FIG. 1A or may include:

The obtaining unit 1501 is configured to obtain at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks of the current image block.

The processing unit 1502 is configured to obtain a probability vector of the current image block according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks and the prediction mode probability model, and multiple elements in the probability vector are related to multiple intra-frame values. The prediction modes are in one-to-one correspondence, and any element in the probability vector is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block.

The determining unit 1503 is configured to determine the target intra prediction mode of the current image block.

The encoding unit 1504 is configured to encode the target intra-frame prediction mode into the code stream according to the target intra-frame prediction mode and the probability vector.

In a possible implementation manner, the above processing unit is specifically configured to: when the target intra prediction mode is the non-matrix weighted intra prediction MIP mode, according to at least one of the reconstruction block, prediction block and residual block of the surrounding image blocks Two, the current image block and the prediction mode probability model to obtain the probability vector of the current image block.

In a possible implementation manner, the above processing unit is specifically configured to: splicing or concatenating at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks to obtain the first data block; The data block and prediction mode probability model obtains the probability vector of the current image block.

In a possible implementation manner, in the aspect of obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model, the processing unit is specifically configured to: when the size of the first data block is the target size, The first data block is input into the prediction mode probability model for processing to obtain the probability vector of the current image block, or; when the size of the first data block is not the target size, a size transformation operation is performed on the first data block to obtain the second data block. data block, the size of the second data block is equal to the target size, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling or equal scaling in both horizontal and vertical directions; The second data block is input into the prediction mode probability model for processing to obtain the probability vector of the current image block.

In a possible implementation manner, in the aspect of obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model, the processing unit is specifically configured to: when the size of the first data block is the target size, The first data block is input into the prediction mode probability model for processing to obtain the probability vector of the current image block, or; when the size of the first data block is not the target size, according to the length of the first side and the second side of the first data block long size relationship, and/or the horizontal gradient and vertical gradient of the current image block, perform a size transformation operation on the first data block to obtain a second data block, the size of the second data block is the target size, and the size transformation operation includes scaling and at least one of transposition, the scaling includes horizontal scaling, vertical scaling or equal scaling in both horizontal and vertical directions; input the second data block into the prediction mode probability model for processing to obtain the current image block probability vector.

In a possible implementation manner, the size transformation operation includes scaling, and according to the size relationship between the length of the first side and the length of the second side of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, for the first data block In the aspect of performing a size transformation operation on a data block to obtain a second data block, the processing unit is specifically used for:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset threshold, the first data block scaling to obtain a second data block, the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the preset threshold, the first The data block is scaled to obtain a second data block, and the first side length and the second side length of the second data block are 8M/N, 8 respectively;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset threshold, the first data block is scaled , to obtain the second data block, the first side length and the second side length of the second data block are respectively 4, 4N/M;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the preset threshold scaling to obtain a second data block, the first side length and the second side length of the second data block are respectively 8,8N/M;

where M and N are positive integers.

In a possible implementation manner, the size transformation operation includes at least one of scaling and transposing, and the size relationship between the first side length and the second side length of the first data block, and/or the size relationship of the current image block For the horizontal gradient and vertical gradient, the size transformation operation is performed on the first data block to obtain the aspect of the second data block, and the processing unit is specifically used for:

When the first side length M of the first data block is smaller than the second side length N of the first data block, perform the following operations on the first data block to obtain the second data block:

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is smaller than the preset threshold, transpose and scale the first data block to obtain a second data block, the first The side length and the second side length are 4N/M, 4 respectively;

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is greater than or equal to the preset threshold, the first data block is transposed and scaled to obtain a second data block whose The length of the first side and the length of the second side are 8N/M, 8 respectively;

where M and N are positive integers.

In a possible implementation manner, the above encoding unit is specifically used to: when the size transformation operation does not include transposition, determine the first reference value according to the first identifier and probability vector of the target intra prediction mode; The value is encoded to obtain the code stream corresponding to the target intra prediction mode;

When the size transformation operation includes transposition, the second identifier is determined according to the preset constant and the first identifier, and the second reference value is determined according to the second identifier and the probability vector; the second reference value is encoded to obtain the target intra prediction The code stream corresponding to the mode.

The above is the process in which the encoding apparatus 1500 specifically performs the method embodiment described in FIG. 13 ; similarly, the encoding apparatus 1500 can also be used to implement all or any steps in the method embodiment shown in FIG. 8 , which is not repeated here. Repeat.

Please refer to FIG. 16, which is a schematic block diagram of an encoding apparatus 1600 provided by the present application. It should be understood that the encoding apparatus 1600 here may correspond to FIG. 1A or the decoder 30 in FIG. 3 , and the encoding apparatus 1600 may include:

The obtaining unit 1601 is configured to obtain a code stream corresponding to the current image block and at least two of the reconstruction blocks, prediction blocks and residual blocks of surrounding image blocks of the current image block.

The processing unit 1602 is configured to obtain a probability vector of the current image block according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks and the prediction mode probability model, and multiple elements in the probability vector are related to multiple intra-frame values. The prediction modes are in one-to-one correspondence, and any element in the probability vector is used to represent the probability of using the intra prediction mode corresponding to any element when predicting the current image block.

The decoding unit 1603 is configured to determine the target intra prediction mode of the current image block according to the code stream and the probability vector, and determine the prediction block of the current image block according to the target intra prediction mode.

In a possible implementation manner, in the aspect of obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model, the processing unit is specifically configured to: when the size of the first data block is the target size, The first data block is input into the prediction mode probability model for processing to obtain the probability vector of the current image block, or;

In a possible implementation manner, the size transformation operation includes scaling. In the above, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, to In the aspect of performing a size transformation operation on the first data block to obtain the second data block, the processing unit is specifically used for:

where M and N are positive integers.

In a possible implementation manner, the size transformation operation includes at least one of scaling and transposition, where the size relationship between the first side length and the second side length of the first data block, and/or the current image block The horizontal gradient and vertical gradient of , perform a size transformation operation on the first data block to obtain the aspect of the second data block, and the processing unit is specifically used for:

where M and N are positive integers.

In a possible implementation manner, the above-mentioned decoding unit is specifically used to: decode the code stream to obtain the target reference value; when the size transformation operation does not include transposition, determine the target intra-frame prediction mode according to the target reference value and the probability vector ;

When the size transformation operation includes transposition, the first identifier is determined according to the target reference value and the probability vector; when the intra-frame prediction mode corresponding to the first identifier is the angle prediction mode, the target intra-frame prediction mode is determined according to the preset constant and the first identifier Prediction mode; when the intra-frame prediction mode corresponding to the first identifier is a non-angle prediction mode, the target intra-frame prediction mode is determined according to the target reference value and the probability vector.

The above is the process in which the encoding apparatus 1600 specifically performs the method embodiment described in FIG. 13 ; similarly, the encoding apparatus 1600 can also be used to implement all or any steps in the method embodiment described in FIG. 14 , which is not repeated here. Repeat.

Those skilled in the art will appreciate that the functions described in connection with the various illustrative logical blocks, modules, and algorithm steps described in this disclosure may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described by the various illustrative logical blocks, modules, and steps may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another (eg, according to a communication protocol) . In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementing the techniques described in this application. The computer program product may comprise a computer-readable medium.

By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory or may be used to store instructions or data structures desired program code in the form of any other medium that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if coaxial cable, fiber optic cable, twisted pair, digital subscriber line DSL, or wireless technologies such as infrared, radio, and microwave are used to transmit instructions from a website, server, or other remote source, coaxial cable, Fiber optic cables, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of media. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc CD, laser disc, optical disc, digital versatile disc DVD, and blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors such as one or more digital signal processors DSPs, general purpose microprocessors, application specific integrated circuits ASICs, field programmable logic arrays FPGAs or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor," as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functions described by the various illustrative logical blocks, modules, and steps described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or in combination with into the combined codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this application may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (eg, a chip set). Various components, modules, or units are described herein to emphasize functional aspects of means for performing the disclosed techniques, but do not necessarily require realization by different hardware units. Indeed, as described above, the various units may be combined in codec hardware units in conjunction with suitable software and/or firmware, or by interoperating hardware units (including one or more processors as described above) supply.

The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited to this. Substitutions should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

A method for encoding an intra-frame prediction mode, wherein the method comprises:

obtaining at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks of the current image block;

A probability vector of the current image block is obtained according to at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks and a prediction mode probability model, and multiple elements in the probability vector are related to multiple intra-frame The prediction modes are in one-to-one correspondence, and any element in the probability vector is used to represent the probability of using the intra prediction mode corresponding to the any element when predicting the current image block;

determining the target intra prediction mode of the current image block;

The target intra-frame prediction mode is encoded into the code stream according to the target intra-frame prediction mode and the probability vector.
The method according to claim 1, wherein the probability vector of the current image block is obtained according to at least two of a reconstruction block, a prediction block and a residual block of the surrounding image blocks and a prediction mode probability model ,include:

When the target intra prediction mode is a non-matrix weighted intra prediction MIP mode, obtain the prediction mode probability model according to at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks The probability vector of the current image patch.
The method according to claim 1 or 2, wherein, obtaining the current image block according to at least two of a reconstruction block, a prediction block and a residual block of the surrounding image blocks and a prediction mode probability model probability vector, including:

splicing or concatenating at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks to obtain a first data block;

The probability vector of the current image block is obtained according to the first data block and the prediction mode probability model.
The method according to claim 3, wherein the obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model comprises:

When the size of the first data block is the target size, inputting the first data block into the prediction mode probability model for processing to obtain a probability vector of the current image block, or;

When the size of the first data block is not the target size, a size transformation operation is performed on the first data block to obtain a second data block, and the size of the second data block is equal to the target size, The size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or proportional scaling in both horizontal and vertical directions; inputting the second data block into all The prediction mode probability model is processed to obtain the probability vector of the current image block.
The method according to claim 3, wherein the obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model comprises:

When the size of the first data block is the target size, inputting the first data block into the prediction mode probability model for processing to obtain a probability vector of the current image block, or;

When the size of the first data block is not the target size, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient sum of the current image block vertical gradient, performing a size transformation operation on the first data block to obtain a second data block, where the size of the second data block is the target size, and the size transformation operation includes at least one of scaling and transposition item, the scaling includes horizontal scaling, vertical scaling or equal scaling in both horizontal and vertical directions; inputting the second data block into the prediction mode probability model for processing to obtain the current image The probability vector for the block.
The method according to claim 5, wherein the size transformation operation comprises the scaling, the scaling according to the size relationship between the first side length and the second side length of the first data block, and/or the The horizontal gradient and vertical gradient of the current image block, and performing a size transformation operation on the first data block to obtain a second data block, including:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset value When the threshold value, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4, 4N/M;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the When the threshold is preset, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8 and 8N/M;

Wherein, the M and the N are positive integers.
The method according to claim 4, wherein the size transformation operation includes at least one of the scaling and the transposing, and the first side length of the first data block and the second The size relationship between the side lengths, and/or the horizontal gradient and vertical gradient of the current image block, and performing a size transformation operation on the first data block to obtain a second data block, including:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, perform the following operations on the first data block to obtain the second data block:

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is less than the preset threshold, performing the transposing and the scaling on the first data block, To obtain the second data block, the first side length and the second side length of the second data block are respectively 4N/M, 4;

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is greater than or equal to the preset threshold, performing the transposing and the transposing on the first data block scaling to obtain the second data block, the first side length and the second side length of the second data block are 8N/M, 8 respectively;

Wherein, the M and the N are positive integers.
The method according to any one of claims 4-7, wherein the encoding the target intra-frame prediction mode into a code stream according to the target intra-frame prediction mode and the probability vector comprises:

When the size transformation operation does not include the transposition, a first reference value is determined according to the first identifier of the target intra-frame prediction mode and the probability vector; and the first reference value is encoded to obtain the code stream corresponding to the target intra prediction mode;

When the size transformation operation includes the transposition and the target intra-frame prediction mode is the angle prediction mode, a second identifier is determined according to a preset constant and the first identifier of the target intra-frame prediction mode, and the second identifier is determined according to the second identifier of the target intra-frame prediction mode. identifying the probability vector and determining a second reference value; encoding the second reference value to obtain a code stream corresponding to the target intra-frame prediction mode; when the size transformation operation includes the transposition, and When the target intra-frame prediction mode is a non-angle prediction mode, a first reference value is determined according to the first identifier of the target intra-frame prediction mode and the probability vector; and the first reference value is encoded to obtain the The code stream corresponding to the target intra prediction mode.
A method for encoding an intra-frame prediction mode, wherein the method comprises:

Get the surrounding image blocks of the current image block;

splicing or concatenating the surrounding image blocks to obtain a first data block;

A size transformation operation is performed on the first data block to obtain a second data block, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or horizontal and vertical scaling. Proportional scaling in both vertical directions;

Inputting the second data block into a prediction mode probability model for processing to obtain a probability vector of the current image block, where multiple elements in the probability vector correspond to multiple intra prediction modes one-to-one, and the probability vector Any element in is used to represent the probability of adopting the intra prediction mode corresponding to the any element when predicting the current image block;

determining the target intra prediction mode of the current image block;

The target intra-frame prediction mode is encoded into the code stream according to the target intra-frame prediction mode and the probability vector.
The method according to claim 9, wherein, performing a size transformation operation on the first data block to obtain a second data block, comprising:

When the target intra prediction mode is a non-matrix weighted intra prediction MIP mode and the size of the first data block is not the target size, the size transformation operation is performed on the first data block to obtain the the second data block, the size of the second data block is equal to the target size.
The method according to claim 9, wherein, performing a size transformation operation on the first data block to obtain a second data block, comprising:

When the target intra prediction mode is the non-matrix weighted intra prediction MIP mode, and the size of the first data block is not the target size, according to the first side length and the second side length of the first data block size relationship, and/or the horizontal gradient and vertical gradient of the current image block, perform the size transformation operation on the first data block to obtain the second data block, the size of the second data block equal to the target size.
The method according to claim 10 or 11, wherein the method further comprises:

When the target intra prediction mode is a non-matrix weighted intra prediction MIP mode and the size of the first data block is equal to the target size, the first data block is input into the prediction mode probability model for processing to Obtain the probability vector of the current image block.
The method according to claim 11 or 12, wherein the size transformation operation includes the scaling, the size relationship according to the first side length and the second side length of the first data block, and/ or the horizontal gradient and vertical gradient of the current image block, performing the size transformation operation on the first data block to obtain the second data block, including:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4, 4N/M;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the When the threshold is preset, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8 and 8N/M;

Wherein, the M and the N are positive integers.
The method according to claim 11 or 12, wherein the size transformation operation includes at least one of the scaling and the transposing, and the first side length of the first data block and the The size relationship of the second side length, and/or the horizontal gradient and the vertical gradient of the current image block, and performing the size transformation operation on the first data block to obtain the second data block, including:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, perform the following operations on the first data block to obtain the second data block:

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is less than the preset threshold, performing the transposing and the scaling on the first data block, To obtain the second data block, the first side length and the second side length of the second data block are respectively 4N/M, 4;

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is greater than or equal to the preset threshold, performing the transposing and the transposing on the first data block scaling to obtain the second data block, the first side length and the second side length of the second data block are 8N/M, 8 respectively;

Wherein, the M and the N are positive integers.
The method according to any one of claims 9-14, wherein the encoding the target intra-frame prediction mode into a code stream according to the target intra-frame prediction mode and the probability vector comprises:

When the size transformation operation does not include the transposition, a first reference value is determined according to the first identifier of the target intra-frame prediction mode and the probability vector; and the first reference value is encoded to obtain the code stream corresponding to the target intra prediction mode;

When the size transformation operation includes the transposition and the target intra-frame prediction mode is the angle prediction mode, a second identifier is determined according to a preset constant and the first identifier of the target intra-frame prediction mode, and the second identifier is determined according to the second identifier of the target intra-frame prediction mode. identifying the probability vector and determining a second reference value; encoding the second reference value to obtain a code stream corresponding to the target intra-frame prediction mode; when the size transformation operation includes the transposition, and When the target intra-frame prediction mode is a non-angle prediction mode, a first reference value is determined according to the first identifier of the target intra-frame prediction mode and the probability vector; and the first reference value is encoded to obtain the The code stream corresponding to the target intra prediction mode.
A decoding method for intra prediction mode, characterized in that the method comprises:

Obtain the code stream corresponding to the current image block, and at least two of the reconstruction blocks, prediction blocks and residual blocks of the surrounding image blocks of the current image block;

A probability vector of the current image block is obtained according to at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks and a prediction mode probability model, and multiple elements in the probability vector are related to multiple intra-frame The prediction modes are in one-to-one correspondence, and any element in the probability vector is used to represent the probability of using the intra prediction mode corresponding to the any element when predicting the current image block;

A target intra prediction mode of the current image block is determined according to the code stream and the probability vector, and a prediction block of the current image block is determined according to the target intra prediction mode.
The method according to claim 16, wherein the probability vector of the current image block is obtained according to at least two of a reconstruction block, a prediction block and a residual block of the surrounding image blocks and a prediction mode probability model ,include:

splicing or concatenating at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks to obtain a first data block;

The probability vector of the current image block is obtained according to the first data block and the prediction mode probability model.
The method according to claim 17, wherein the obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model comprises:

When the size of the first data block is the target size, inputting the first data block into the prediction mode probability model for processing to obtain a probability vector of the current image block, or;

When the size of the first data block is not the target size, a size transformation operation is performed on the first data block to obtain a second data block, and the size of the second data block is equal to the target size, The size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or proportional scaling in both horizontal and vertical directions; inputting the second data block into all The prediction mode probability model is processed to obtain the probability vector of the current image block.
The method according to claim 17, wherein the obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model comprises:

When the size of the first data block is the target size, the first data block is input into the prediction mode probability model for processing to obtain the probability vector of the current image block, or;

When the size of the first data block is not the target size, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient sum of the current image block vertical gradient, performing a size transformation operation on the first data block to obtain a second data block, where the first side length and the second side length are the difference between two mutually perpendicular sides in the first data block length, the size of the second data block is the target size, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or both horizontal and vertical scaling proportional scaling in each direction; inputting the second data block into the prediction mode probability model for processing to obtain the probability vector of the current image block.
The method according to claim 19, wherein the size transformation operation comprises the scaling, the size relationship according to the first side length and the second side length of the first data block, and/or the For the horizontal gradient and vertical gradient of the current image block, a size transformation operation is performed on the first data block to obtain a second data block, including:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4, 4N/M;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the When the threshold is preset, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8 and 8N/M;

Wherein, the M and the N are positive integers.
The method according to claim 19, wherein the size transformation operation includes at least one of the scaling and the transposing, and the first side length of the first data block and the second The size relationship between the side lengths, and/or the horizontal gradient and vertical gradient of the current image block, and performing a size transformation operation on the first data block to obtain a second data block, including:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, perform the following operations on the first data block to obtain the second data block:

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is less than the preset threshold, performing the transposing and the scaling on the first data block, To obtain the second data block, the first side length and the second side length of the second data block are respectively 4N/M, 4;

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is greater than or equal to the preset threshold, performing the transposing and the transposing on the first data block scaling to obtain the second data block, the first side length and the second side length of the second data block are 8N/M, 8 respectively;

Wherein, the M and the N are positive integers.
The method according to any one of claims 18-21, wherein the determining the target intra prediction mode of the current image block according to the code stream and the probability vector comprises:

Decoding the code stream to obtain a target reference value;

When the size transformation operation does not include the transposition, determining the target intra prediction mode according to the target reference value and the probability vector;

When the size transformation operation includes the transposition, a first identifier is determined according to the target reference value and the probability vector; when the intra prediction mode corresponding to the first identifier is an angle prediction mode, a preset The constant and the first identifier are used to determine the target intra-frame prediction mode; when the intra-frame prediction mode corresponding to the first identifier is a non-angular prediction mode, the target reference value and the probability vector are determined according to the Target intra prediction mode.
A decoding method for intra prediction mode, characterized in that the method comprises:

Obtain the code stream corresponding to the current image block, and the surrounding image blocks of the current image block;

splicing or concatenating the surrounding image blocks to obtain a first data block;

A size transformation operation is performed on the first data block to obtain a second data block, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or horizontal and vertical scaling. Proportional scaling in both vertical directions;

Inputting the second data block into a prediction mode probability model for processing to obtain a probability vector of the current image block, where multiple elements in the probability vector correspond to multiple intra prediction modes one-to-one, and the probability vector Any element in is used to represent the probability of adopting the intra prediction mode corresponding to the any element when predicting the current image block;

A target intra prediction mode of the current image block is determined according to the code stream and the probability vector, and a prediction block of the current image block is determined according to the target intra prediction mode.
The method according to claim 23, wherein, performing a size transformation operation on the first data block to obtain a second data block, comprising:

When the size of the first data block is not the target size, the size transformation operation is performed on the first data block to obtain the second data block, and the size of the second data block is equal to the target size size.
The method according to claim 23, wherein, performing a size transformation operation on the first data block to obtain a second data block, comprising:

When the size of the first data block is not the target size, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block , performing the size transformation operation on the first data block to obtain the second data block, where the size of the second data block is equal to the target size.
The method according to claim 24 or 25, wherein the method further comprises:

When the size of the first data block is equal to the target size, the first data block is input into the prediction mode probability model for processing to obtain a probability vector of the current image block.
The method according to claim 25 or 26, wherein the size transformation operation comprises the scaling, the size relationship according to the first side length and the second side length of the first data block, and/or The horizontal gradient and vertical gradient of the current image block, and performing the size transformation operation on the first data block to obtain the second data block, including:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4, 4N/M;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the When the threshold is preset, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8 and 8N/M;

Wherein, the M and the N are positive integers.
The method according to claim 25 or 26, wherein the size transformation operation includes at least one of the scaling and the transposing, and the first side length of the first data block and the The size relationship of the second side length, and/or the horizontal gradient and the vertical gradient of the current image block, and performing the size transformation operation on the first data block to obtain the second data block, including:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, perform the following operations on the first data block to obtain the second data block:

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is less than the preset threshold, performing the transposing and the scaling on the first data block, To obtain the second data block, the first side length and the second side length of the second data block are respectively 4N/M, 4;

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is greater than or equal to the preset threshold, performing the transposing and the transposing on the first data block scaling to obtain the second data block, the first side length and the second side length of the second data block are 8N/M, 8 respectively;

Wherein, the M and the N are positive integers.
The method according to any one of claims 23-28, wherein the determining the target intra prediction mode of the current image block according to the code stream and the probability vector comprises:

Decoding the code stream to obtain a target reference value;

When the size transformation operation does not include the transposition, determining the target intra prediction mode according to the target reference value and the probability vector;

When the size transformation operation includes the transposition, a first identifier is determined according to the target reference value and the probability vector; when the intra prediction mode corresponding to the first identifier is an angle prediction mode, a preset The constant and the first identifier are used to determine the target intra-frame prediction mode; when the intra-frame prediction mode corresponding to the first identifier is a non-angular prediction mode, the target reference value and the probability vector are determined according to the Target intra prediction mode.
An encoding device, characterized in that the device comprises:

an acquisition unit for acquiring at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks of the current image block;

A processing unit, configured to obtain a probability vector of the current image block according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image block and the prediction mode probability model, and a plurality of elements in the probability vector One-to-one correspondence with multiple intra-frame prediction modes, and any element in the probability vector is used to represent the probability of using the intra-frame prediction mode corresponding to the any element when predicting the current image block;

a determining unit for determining a target intra prediction mode of the current image block;

A coding unit, configured to encode the target intra-frame prediction mode into a code stream according to the target intra-frame prediction mode and the probability vector.
The device according to claim 30, wherein the processing unit is specifically configured to:

When the target intra prediction mode is a non-matrix weighted intra prediction MIP mode, the probability of the current image block and the prediction mode is determined according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image blocks. The model obtains the probability vector of the current image block.
The device according to claim 30 or 31, wherein the processing unit is specifically configured to:

splicing or concatenating at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks to obtain a first data block;

The probability vector of the current image block is obtained according to the first data block and the prediction mode probability model.
The apparatus according to claim 32, wherein, in the aspect of obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model, the processing unit is specifically configured to: :

When the size of the first data block is the target size, inputting the first data block into the prediction mode probability model for processing to obtain a probability vector of the current image block, or;

When the size of the first data block is not the target size, a size transformation operation is performed on the first data block to obtain a second data block, and the size of the second data block is equal to the target size, The size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or proportional scaling in both horizontal and vertical directions; inputting the second data block into all The prediction mode probability model is processed to obtain the probability vector of the current image block.
The apparatus according to claim 32, wherein, in the aspect of obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model, the processing unit is specifically configured to: :

When the size of the first data block is the target size, inputting the first data block into the prediction mode probability model for processing to obtain a probability vector of the current image block, or;

When the size of the first data block is not the target size, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient sum of the current image block vertical gradient, performing a size transformation operation on the first data block to obtain a second data block, where the size of the second data block is the target size, and the size transformation operation includes at least one of scaling and transposition item, the scaling includes horizontal scaling, vertical scaling or equal scaling in both horizontal and vertical directions; inputting the second data block into the prediction mode probability model for processing to obtain the current image The probability vector for the block.
The apparatus according to claim 34, wherein the size transformation operation includes the scaling, in the said first data block according to the size relationship between the first side length and the second side length, and/ Or the horizontal gradient and vertical gradient of the current image block, performing a size transformation operation on the first data block to obtain the aspect of the second data block, and the processing unit is specifically used for:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4, 4N/M;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the When the threshold is preset, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8 and 8N/M;

Wherein, the M and the N are positive integers.
The apparatus according to claim 34, wherein the size transformation operation includes at least one of the scaling and the transposing, and wherein the length of the first side of the first data block is different from that of the first data block. The size relationship of the second side length, and/or the horizontal gradient and the vertical gradient of the current image block, the size transformation operation is performed on the first data block to obtain the second data block, the processing unit specifically uses At:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, perform the following operations on the first data block to obtain the second data block:

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is less than the preset threshold, performing the transposing and the scaling on the first data block, To obtain the second data block, the first side length and the second side length of the second data block are respectively 4N/M, 4;

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is greater than or equal to the preset threshold, performing the transposing and the transposing on the first data block scaling to obtain the second data block, the first side length and the second side length of the second data block are 8N/M, 8 respectively;

Wherein, the M and the N are positive integers.
The device according to any one of claims 33-36, wherein the encoding unit is specifically used for:

When the size transformation operation does not include the transposition, a first reference value is determined according to the first identifier of the target intra-frame prediction mode and the probability vector; and the first reference value is encoded to obtain the code stream corresponding to the target intra prediction mode;

When the size transformation operation includes the transposition, a second identifier is determined according to a preset constant and the first identifier, and a second reference value is determined according to the second identifier and the probability vector; Two reference values are encoded to obtain a code stream corresponding to the target intra-frame prediction mode.
An encoding device, characterized in that the device comprises:

an acquisition unit for acquiring the surrounding image blocks of the current image block;

a processing unit, configured to splicing or concatenating the surrounding image blocks to obtain a first data block; performing a size transformation operation on the first data block to obtain a second data block, the size transformation operation including scaling and at least one of transposition, the scaling includes horizontal scaling, vertical scaling or equal scaling in both horizontal and vertical directions; and for inputting the second data block into a prediction mode probability model for processing , to obtain the probability vector of the current image block, multiple elements in the probability vector are in one-to-one correspondence with multiple intra prediction modes, and any element in the probability vector is used to represent the current image block. The probability of using the intra prediction mode corresponding to any of the elements when performing prediction;

a determining unit for determining a target intra prediction mode of the current image block;

A coding unit, configured to encode the target intra-frame prediction mode into a code stream according to the target intra-frame prediction mode and the probability vector.
The apparatus according to claim 38, wherein, in the aspect of performing a size transformation operation on the first data block to obtain the second data block, the processing unit is specifically configured to:

When the target intra prediction mode is a non-matrix weighted intra prediction MIP mode and the size of the first data block is not the target size, the size transformation operation is performed on the first data block to obtain the the second data block, the size of the second data block is equal to the target size.
The apparatus according to claim 38, wherein, in the aspect of performing a size transformation operation on the first data block to obtain the second data block, the processing unit is specifically configured to:

When the target intra prediction mode is the non-matrix weighted intra prediction MIP mode, and the size of the first data block is not the target size, according to the first side length and the second side length of the first data block size relationship, and/or the horizontal gradient and vertical gradient of the current image block, perform the size transformation operation on the first data block to obtain the second data block, the size of the second data block equal to the target size.
The apparatus according to claim 39 or 40, wherein the processing unit is further configured to:

When the target intra prediction mode is a non-matrix weighted intra prediction MIP mode and the size of the first data block is equal to the target size, the first data block is input into the prediction mode probability model for processing to Obtain the probability vector of the current image block.
The apparatus according to claim 40 or 41, wherein the size transformation operation includes the scaling, and in the size relationship according to the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, performing the size transformation operation on the first data block to obtain the aspect of the second data block, the processing unit is specifically used for:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4, 4N/M;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the When the threshold is preset, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8 and 8N/M;

Wherein, the M and the N are positive integers.
The apparatus according to claim 40 or 41, wherein the size transformation operation includes the scaling, and in the size relationship according to the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block, performing the size transformation operation on the first data block to obtain the aspect of the second data block, the processing unit is specifically used for:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, perform the following operations on the first data block to obtain the second data block:

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is less than the preset threshold, performing the transposing and the scaling on the first data block, To obtain the second data block, the first side length and the second side length of the second data block are respectively 4N/M, 4;

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is greater than or equal to the preset threshold, performing the transposing and the transposing on the first data block scaling to obtain the second data block, the first side length and the second side length of the second data block are 8N/M, 8 respectively;

Wherein, the M and the N are positive integers.
The device according to any one of claims 38-41, wherein the encoding unit is specifically used for:

When the size transformation operation does not include the transposition, a first reference value is determined according to the first identifier of the target intra-frame prediction mode and the probability vector; and the first reference value is encoded to obtain the code stream corresponding to the target intra prediction mode;

When the size transformation operation includes the transposition, a second identifier is determined according to a preset constant and the first identifier, and a second reference value is determined according to the second identifier and the probability vector; Two reference values are encoded to obtain a code stream corresponding to the target intra-frame prediction mode.
A decoding device, characterized in that the device comprises:

an acquisition unit, configured to acquire a code stream corresponding to the current image block, and at least two of the reconstruction blocks, prediction blocks and residual blocks of the surrounding image blocks of the current image block;

A processing unit, configured to obtain a probability vector of the current image block according to at least two of the reconstruction block, the prediction block and the residual block of the surrounding image block and the prediction mode probability model, and a plurality of elements in the probability vector One-to-one correspondence with multiple intra-frame prediction modes, and any element in the probability vector is used to represent the probability of using the intra-frame prediction mode corresponding to the any element when predicting the current image block;

A decoding unit, configured to determine a target intra prediction mode of the current image block according to the code stream and the probability vector, and determine a prediction block of the current image block according to the target intra prediction mode.
The apparatus according to claim 45, wherein the processing unit is specifically configured to:

splicing or concatenating at least two of the reconstruction block, prediction block and residual block of the surrounding image blocks to obtain a first data block;

The probability vector of the current image block is obtained according to the first data block and the prediction mode probability model.
The apparatus according to claim 46, wherein, in the aspect of obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model, the processing unit is specifically configured to:

When the size of the first data block is the target size, inputting the first data block into the prediction mode probability model for processing to obtain a probability vector of the current image block, or;

When the size of the first data block is not the target size, a size transformation operation is performed on the first data block to obtain a second data block, and the size of the second data block is equal to the target size, The size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or proportional scaling in both horizontal and vertical directions; inputting the second data block into all The prediction mode probability model is processed to obtain the probability vector of the current image block.
The apparatus according to claim 46, wherein, in the aspect of obtaining the probability vector of the current image block according to the first data block and the prediction mode probability model, the processing unit is specifically configured to:

When the size of the first data block is the target size, inputting the first data block into the prediction mode probability model for processing to obtain a probability vector of the current image block, or;

When the size of the first data block is not the target size, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient sum of the current image block vertical gradient, performing a size transformation operation on the first data block to obtain a second data block, where the first side length and the second side length are the difference between two mutually perpendicular sides in the first data block length, the size of the second data block is the target size, the size transformation operation includes at least one of scaling and transposition, and the scaling includes horizontal scaling, vertical scaling, or both horizontal and vertical scaling proportional scaling in each direction; inputting the second data block into the prediction mode probability model for processing to obtain the probability vector of the current image block.
The apparatus according to claim 48, wherein the size transformation operation comprises the scaling, in the size relationship according to the first side length and the second side length of the first data block, and/or For the horizontal gradient and vertical gradient of the current image block, a size transformation operation is performed on the first data block to obtain the second data block, and the processing unit is specifically used for:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4, 4N/M;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the When the threshold is preset, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8 and 8N/M;

Wherein, the M and the N are positive integers.
The apparatus according to claim 48, wherein the size transformation operation includes at least one of the scaling and the transposing, and the size conversion operation is based on the first side length of the first data block and the first data block. The size relationship between the two sides, and/or the horizontal gradient and the vertical gradient of the current image block, the size transformation operation is performed on the first data block to obtain the second data block, the processing unit is specifically used for :

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, the following operations are performed on the first data block to obtain the second data block:

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is less than the preset threshold, performing the transposing and the scaling on the first data block, To obtain the second data block, the first side length and the second side length of the second data block are respectively 4N/M, 4;

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is greater than or equal to the preset threshold, performing the transposing and the transposing on the first data block scaling to obtain the second data block, the first side length and the second side length of the second data block are 8N/M, 8 respectively;

Wherein, the M and the N are positive integers.
The apparatus according to claims 47-50, wherein the decoding unit is specifically configured to:

Decoding the code stream to obtain a target reference value;

When the size transformation operation does not include the transposition, determining the target intra prediction mode according to the target reference value and the probability vector;

When the size transformation operation includes the transposition, a first identifier is determined according to the target reference value and the probability vector; when the intra prediction mode corresponding to the first identifier is an angle prediction mode, a preset The constant and the first identifier are used to determine the target intra-frame prediction mode; when the intra-frame prediction mode corresponding to the first identifier is a non-angular prediction mode, the target reference value and the probability vector are determined to determine the target intra-frame prediction mode. Target intra prediction mode.
A decoding device, characterized in that the device comprises:

an acquisition unit, used for acquiring the code stream corresponding to the current image block, and the surrounding image blocks of the current image block;

a processing unit, configured to splicing or concatenating the surrounding image blocks to obtain a first data block; performing a size transformation operation on the first data block to obtain a second data block, the size transformation operation including scaling and at least one of transposition, the scaling includes horizontal scaling, vertical scaling or equal scaling in both horizontal and vertical directions; and for inputting the second data block into a prediction mode probability model for processing , to obtain the probability vector of the current image block, multiple elements in the probability vector are in one-to-one correspondence with multiple intra prediction modes, and any element in the probability vector is used to represent the current image block. The probability of using the intra prediction mode corresponding to any of the elements when performing prediction;

A decoding unit, configured to determine a target intra prediction mode of the current image block according to the code stream and the probability vector, and determine a prediction block of the current image block according to the target intra prediction mode.
The apparatus according to claim 52, wherein, in the aspect of performing a size transformation operation on the first data block to obtain a second data block, the processing unit is specifically configured to:

When the size of the first data block is not the target size, the size transformation operation is performed on the first data block to obtain the second data block, and the size of the second data block is equal to the target size size.
The apparatus according to claim 52, wherein, in the aspect of performing a size transformation operation on the first data block to obtain a second data block, the processing unit is specifically configured to:

When the size of the first data block is not the target size, according to the size relationship between the first side length and the second side length of the first data block, and/or the horizontal gradient and vertical gradient of the current image block , performing the size transformation operation on the first data block to obtain the second data block, where the size of the second data block is equal to the target size.
The apparatus according to claim 53 or 54, wherein the processing unit is further configured to:

When the size of the first data block is equal to the target size, the first data block is input into the prediction mode probability model for processing to obtain a probability vector of the current image block.
The apparatus according to claim 54 or 55, wherein the size transformation operation includes the scaling, in the size relationship according to the first side length and the second side length of the first data block, and / or the horizontal gradient and vertical gradient of the current image block, performing a size transformation operation on the first data block to obtain the aspect of the second data block, the processing unit is specifically used for:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than the preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4, 4N/M;

When the first side length M of the first data block is smaller than the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to the When the threshold is preset, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8 and 8N/M;

Wherein, the M and the N are positive integers.
The apparatus according to claim 54 or 55, wherein the size transformation operation includes at least one of the scaling and the transposing, and in the first data block according to the first side length The size relationship with the second side length, and/or the horizontal gradient and vertical gradient of the current image block, the size transformation operation is performed on the first data block to obtain the aspect of the second data block, the processing unit specifically Used for:

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is less than a preset value When the threshold is set, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 4M/N, 4;

When the first side length M of the first data block is greater than or equal to the second side length N of the first data block, and the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient is greater than or equal to At the preset threshold, the scaling is performed on the first data block to obtain the second data block, and the first side length and the second side length of the second data block are respectively 8M/N, 8;

When the first side length M of the first data block is smaller than the second side length N of the first data block, perform the following operations on the first data block to obtain the second data block:

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is less than the preset threshold, performing the transposing and the scaling on the first data block, To obtain the second data block, the first side length and the second side length of the second data block are respectively 4N/M, 4;

When the sum of the absolute value of the horizontal gradient and the absolute value of the vertical gradient of the first data block is greater than or equal to the preset threshold, performing the transposing and the transposing on the first data block scaling to obtain the second data block, the first side length and the second side length of the second data block are 8N/M, 8 respectively;

Wherein, the M and the N are positive integers.
The apparatus according to claims 52-57, wherein the decoding unit is specifically configured to:

Decoding the code stream to obtain a target reference value;

When the size transformation operation does not include the transposition, determining the target intra prediction mode according to the target reference value and the probability vector;

When the size transformation operation includes the transposition, a first identifier is determined according to the target reference value and the probability vector; when the intra prediction mode corresponding to the first identifier is an angle prediction mode, a preset The constant and the first identifier are used to determine the target intra-frame prediction mode; when the intra-frame prediction mode corresponding to the first identifier is a non-angular prediction mode, the target reference value and the probability vector are determined according to the Target intra prediction mode.
An encoder (20), characterized by comprising a processing circuit for performing the method according to any one of claims 1 to 15.
A decoder (30), characterized by comprising a processing circuit for performing the method according to any one of claims 16 to 29.
A kind of encoder, is characterized in that, comprises:

one or more processors;

A non-transitory computer-readable storage medium coupled to the processor and storing a program for execution by the processor, wherein the program, when executed by the processor, causes the encoder to perform according to claims 1 to The method of any one of 15.
A decoder, characterized in that it includes:

one or more processors;

A non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform a program according to claim 16 to The method of any one of 29.
A computer program product, characterized by comprising program code for carrying out the method according to any one of claims 1 to 29 when executed on a computer or processor.
A non-transitory computer-readable storage medium comprising program code for performing the method according to any one of claims 1 to 29 when executed by a computer device.
A non-transitory storage medium, comprising a bit stream encoded according to the method of any one of claims 1 to 15.