KR102309910B1 - Optimal mode decision unit of video encoder and video encoding method using the optimal mode decision - Google Patents

Abstract

The present invention relates to an apparatus for determining an optimal mode of a video encoder and a video encoding method using the determination of an optimal mode, wherein the apparatus for determining an optimal mode generates a combined image by combining an input image and a surrounding image based on an encoding time. and a mode determining device for determining a mode of a video encoder for a current CTU (Coding Tree Unit) based on a Convolutional Neural Network (CNN) by inputting the corresponding combined image and quantization parameters, and through this, one CTU (Coding The optimal mode can be determined without performing the rate-distortion cost calculation process for all modes that the tree unit can have, and the encoding efficiency is improved.

Description

{Optimal mode decision unit of video encoder and video encoding method using the optimal mode decision}

The present invention relates to a video encoder, and more particularly, an apparatus for determining an optimal mode of a video encoder capable of improving encoding processing efficiency by determining an optimal mode for performing video encoder processing, and a video encoding method using the optimal mode determination is about

Encoders of video standards (MPEG-2/MPEG-4/AVC/HEVC/SVC/SHVC, etc.) established by ISO/IEC and public video standards (XVID/Dirac/Theora/Daala/VP8/VP9, etc.) In terms of providing the maximum picture quality while guaranteeing the bit rate, each has its own encoding tool as well as its own mode determination method. In particular, the HEVC video standard established in January 2013 shows better compression performance than the conventional video standard in that it has an excellent encoding tool and an optimal mode determination method. In particular, compared to the previous standard, AVC, it shows a 30-50% coding compression improvement rate, but has a more complicated problem by 120-200% in terms of computational complexity.

In particular, as video encoding technology is advanced, video standards have adopted various encoding modes in order to increase encoding efficiency, and for this reason, a method of selecting an optimal mode among many encoding modes has also been proposed. Among these methods, a representative method is Rate-Distortion Optimization (RDO), which calculates the cost values of all modes, and selects the mode with the smallest cost value as the optimal mode. a way to decide Since this conventional method is a method of calculating the cost values of all modes, it has a limitation that the amount of calculation increases as the number of modes increases. As the amount of computation increases, the computational complexity of the encoder increases, which has a limiting factor in that it is difficult to utilize in real-time processing that requires processing 30 or 60 frames per second at a high resolution of FHD (Full High Definition) or higher. At this time, since the cost calculation is performed through transform and quantization/inverse quantization and inverse transform/entropy encoding for all modes, (number of modes) x (transform and quantization/inverse quantization and inverse transform/entropy encoding) have to be performed. .

This is because the prior art has been developed in the form of sacrificing computational complexity to improve compression efficiency, and a new method for improving encoding efficiency is required.

Korean Patent Laid-Open Patent No. 10-2009-0040028 (published on April 23, 2009)

An object of the present invention to solve the above problems is an apparatus for determining an optimal mode of a video encoder that determines a mode at high speed while maximally securing encoding compression efficiency using a convolutional neural network (CNN) in a video encoder, and determining the optimal mode An object of the present invention is to provide a video encoding method used.

An apparatus for determining an optimal mode of a video encoder of the present invention for achieving the above object includes an image combining apparatus for generating a combined image by combining an input image and a peripheral image based on an encoding time, and the combined image and quantization parameters It is characterized in that it comprises a mode determining device for determining the mode of the video encoder for the current CTU (Coding Tree Unit) based on CNN (Convolutional Neural Network) as an input.

In the apparatus for determining an optimal mode of a video encoder according to the present invention, the image combining apparatus combines a current block corresponding to an input image and a neighboring block corresponding to a neighboring image and located in the vicinity of the current block, wherein the neighboring block is It is characterized in that it is determined using the CTU of the frame or the pre-processed CTU.

In the optimal mode determining apparatus of a video encoder of the present invention, the mode determining apparatus includes a plurality of mode determining apparatuses corresponding to the size of the CTU, and among the modes output from the plurality of mode determining apparatuses, the rate-distortion cost is It is characterized by selecting a relatively small number of modes or selecting an optimal mode using machine learning classification.

In the video encoding method using the optimal mode determination of the present invention for achieving the above object, a video encoder generates a combined image by combining an input image and a peripheral image based on an encoding time, and the combined image and quantization parameters Determining the mode of the video encoder for the current CTU (Coding Tree Unit) based on CNN (Convolutional Neural Network) as an input, transforming and quantizing the input CTU according to the determined mode by the video encoder, and the and encoding, by the video encoder, transform coefficients obtained through transform and quantization using an entropy coding engine.

In the video encoding method using the optimal mode determination of the present invention, after the transforming and quantizing, the pixels are reconstructed through inverse quantization and inverse transform processes, the reconstructed pixels are subjected to deblocking filtering and sample adaptive offset (sample adaptive offset). ) through a filtering process and storing in a Decoded Picture Buffer (DPB), and a difference value from a prediction block obtained through motion compensation and intra compensation for predicting motion based on the frame stored in the DPB. It characterized in that it further comprises the step of utilizing as an input for transformation and quantization.

In the video encoding method using the optimal mode determination of the present invention, the steps of transforming and quantizing and encoding for a specific CTU are performed separately from the step of determining the mode of a video encoder for another CTU. .

According to the apparatus for determining the optimal mode of a video encoder and the video encoding method using the optimal mode determination of the present invention, since it directly determines which class the input image is in for a given input image, one Coding Tree Unit (CTU) ), it is possible to determine the optimal mode without performing the rate-distortion cost calculation process for all modes.

In addition, since CNN benefits significantly in terms of operation speed when using a GPU (Graphic Processing Unit), it performs operations in parallel with other encoding operations based on the operation of the CPU (Central Processing Unit), resulting in the total amount of encoding operation. is greatly reduced, enabling high-speed encoding.

1 is a flowchart of an operation of a video encoder according to an embodiment of the present invention.
2 is a diagram illustrating an apparatus for determining an optimal mode according to an embodiment of the present invention.
3 is a diagram illustrating an input configuration of an image combining apparatus according to an embodiment of the present invention.
4 is a diagram illustrating an input configuration of an image combining apparatus according to another embodiment of the present invention.
5 is a diagram illustrating an input configuration of the image combining apparatus according to the embodiment of FIG. 4 .
6 is a diagram illustrating an apparatus for determining an optimal mode according to another embodiment of the present invention.
7 is an exemplary diagram illustrating an operation of a mode determining apparatus according to an embodiment of the present invention.
8 is a diagram illustrating a state in which optimal mode determination and encoding are performed separately according to an embodiment of the present invention.
9 is a flowchart illustrating a process of a video encoding method according to an embodiment of the present invention.

It should be noted that, in the following description, only parts necessary for understanding the embodiments of the present invention are described, and descriptions of other parts will be omitted so as not to obscure the gist of the present invention.

The terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors have appropriate concepts of terms in order to best describe their inventions. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined in Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only preferred embodiments of the present invention, and do not represent all of the technical spirit of the present invention, so various equivalents that can be substituted for them at the time of the present application It should be understood that there may be variations and variations.

The present invention relates to mode determination of a video encoder. Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

1 is a flowchart of an operation of a video encoder according to an embodiment of the present invention.

Referring to FIG. 1 , the video encoder performs encoding in units of specific blocks in consideration of the arithmetic processing capability of a memory or hardware. At this time, encoding is performed by dividing one frame into several coding tree units (CTUs), and the optimal mode determining apparatus 100, 200 includes a residual CTU (CTU) of an input frame, a CTU of a previous frame, or a pre-processed CTU. (Prev. Picture CTU) and consider the quantization parameter (QP) from the quantization controller 1 to determine the mode of the video encoder for the current CTU (Coding Tree Unit) based on CNN (Convolutional Neural Network). decide CNN is one of the feed-forward neural networks in the field of machine learning, and it exhibits excellent recognition performance in the field of voice, image, and video recognition.

And the input for transform and quantization of the CTU becomes a difference value between the input CTU and a prediction block obtained through an intra-compensation or motion compensation process. It passes through an entropy coding engine such as Arithmetic Coding) and is converted into a bitstream composed of 0's and 1's.

At this time, after the transformation and quantization of the CTU, the pixel is restored through inverse quantization and inverse transformation, and the reconstructed pixel is subjected to deblocking filtering and sample adaptive offset filtering, followed by a Decoded Picture Buffer (DPB) (2) save to And the difference value between the prediction block obtained through the motion compensation that predicts motion based on the frame stored in the DPB and the intra compensation process using intra prediction is used as an input for transformation and quantization of the CTU. In this case, intra prediction has a plurality of modes, and optimal motion vectors and modes are found for prediction units (PUs) having various sizes through motion prediction.

In this video encoding process, the bit rate and image quality change depending on the mode selection efficiency of intra prediction and motion prediction or the quantization selection method of the quantization controller.

2 is a diagram illustrating an apparatus 100 for determining an optimal mode according to an embodiment of the present invention.

Referring to FIG. 2 , the optimal mode determining apparatus 100 includes an image combining apparatus 10 and a mode determining apparatus 20 .

The optimal mode determining apparatus 100 is an apparatus that determines an optimal mode when encoding of a CTU starts, and performs encoding accordingly.

The image combining apparatus 10 receives a CTU input of an input image frame and generates a combined image by receiving a CTU input of a neighboring image frame based on an encoding time. In this case, the image combining apparatus 10 combines the current block corresponding to the input image and the neighboring block corresponding to the neighboring image. The neighboring block is located in the vicinity of the current block, and is determined using the CTU of the previous frame or the pre-processed CTU.

The mode determining device 20 determines the mode of the video encoder for the current CTU based on CNN by receiving the combined image output from the image combining device 10 and the quantization parameter (QP) as inputs.

A process of combining images by the image combining apparatus 10 of the optimal mode determining apparatus 100 will be described with reference to FIGS. 3 to 5 .

3 is a diagram illustrating an input configuration of an image combining apparatus according to an embodiment of the present invention.

Referring to FIG. 3 , a combined image is generated by combining a current block corresponding to an input image of an image combining apparatus and a neighboring block. In this case, the neighboring block can be divided into a method using future information after the current block and a method using previously encoded past information, and a combination of images can be selected in consideration of encoding efficiency and compression operation amount.

4 is a diagram illustrating an input configuration of an image combining apparatus according to another embodiment of the present invention, and FIG. 5 is a diagram illustrating an input configuration of an image combining apparatus according to the embodiment of FIG. 4 .

4 and 5 show the configuration of neighboring blocks using the previous frame obtained from the DPB shown in FIG. 1 . At this time, a neighboring block may be constructed using the motion vector information (arrows shown in FIG. 4 ). In this case, the combined image has the same size as the current block and the neighboring block, and the current block of the corresponding size and the neighboring block are added or By subtracting it, you can compose a combined image.

6 is a diagram illustrating an apparatus 200 for determining an optimal mode according to another embodiment of the present invention.

Referring to FIG. 6 , the optimal mode determining apparatus 200 includes an image combining apparatus 11 , a plurality of mode determining apparatuses 21 , and a mode selecting apparatus 30 .

The optimal mode determining apparatus 200 determines the optimal mode when encoding of the CTU starts, and the encoding proceeds accordingly.

The image combining device 11 receives a CTU input of an input image frame and generates a combined image by receiving a CTU input of a neighboring image frame based on an encoding time. In this case, the image combining device 11 combines the current block corresponding to the input image and the neighboring block corresponding to the neighboring image. The neighboring block is located in the vicinity of the current block, and is determined using the CTU of the previous frame or the pre-processed CTU.

The plurality of mode determining unit 21 determines the mode of the video encoder for the current CTU based on CNN by receiving the combined image output from the image combining unit 11 and the quantization parameter (QP) as inputs.

In this case, the plurality of mode determining apparatus 21 determines and outputs the plurality of modes, respectively, according to the size of the combined image. For example, if the combined image has a size of 64x64, mode determination device 0 may be used, if it has a size of 32x32, mode determination device 1 may be used, and if it has a size of 16x16, mode determination device N may be used.

The mode selection device 30 is a device for selecting the most appropriate mode among a plurality of modes output from the plurality of mode determining devices 21 .

Referring to the intra prediction method with respect to the combined image shown in FIG. 3 as an example, the image combining device 11 sends a 64x64 unit combined image to the mode determining device 0, and a 32x32 unit combined image to the mode determining device 1 4 times, and 16x16 unit video is sent to the mode determining device N. And the mode determining device 0 determines the intra-direction of 64x64 units, the mode determining device 1 determines the intra-direction for 4 32x32 unit blocks through 4 repetitions, and the mode determination unit N determines the intra-direction through 16 repetitions. Intra directions for 16 16x16 unit blocks are determined. Thereafter, the mode selection device 30 operates in a manner of selecting an optimal mode from among the modes outputted from the mode determining device 0/1/N. The optimal mode may be selected by calculating the rate-distortion cost and selecting the mode having the smallest value, or selecting the optimal mode through a machine learning classification method or the like.

7 is an exemplary diagram illustrating an operation of a mode determining apparatus according to an embodiment of the present invention.

In the convolutional layers shown on the left of FIG. 7, when a 64x64 sized input image is input as an input of the mode determining device, 12 3x3 convolutional filters are traversed by 3 pixels with respect to the 64x64 sized input image (stride of 3) Apply. The 12 filters are divided into 3 GPUs, 4 each, and the GPU applies convolutional filters as many as the given number of filters. Therefore, one GPU has an image of size 21x21x4 (width x length x number of filters) in the second step. In the next step, the GPU generates a 10x10x2 image by applying two 3x3x4 filters and then selecting the maximum value in the 2x2 pixel area (max-pooling).

The 10x10x2 image finally obtained from the convolutional layers is entered as the input value of the first fully-connected layers operated on three GPUs.

And the output of the first fully-connected layers goes into the input value of the second fully-connected layers operated by one GPU. At this time, the output of the secondary fully-connected layers has a probability value. In this case, the mode having the maximum probability value becomes the final mode. As shown on the right side of FIG. 7 , it is divided into four CUs (coding units) of 32x32 size, and class 2 having an intra prediction mode of 2Nx2N, respectively, is selected as the optimal mode for 64x64 CTU with a probability of 0.820. do.

8 is a diagram illustrating a state in which optimal mode determination and encoding are performed separately according to an embodiment of the present invention.

The encoding process according to the present invention may operate the optimal mode determination process and the encoding process separately, and may be implemented in a form in which the optimal mode determination process is implemented in the encoding process. For this purpose, an example of separately operating the optimal mode determination process and the encoding process is shown.

In FIG. 8 , the apparatus for determining the optimal mode operates separately from an encoding process of generating a bitstream using an input image in units of CTUs in the GPU. At this time, the video encoder checks the optimal mode determined by the optimal mode determining device, and generates a bitstream by performing transformation, quantization, and entropy coding of the CTU in the corresponding mode.

A video encoding process according to the present invention will be described with reference to FIG. 9 .

9 is a flowchart illustrating a process of a video encoding method according to an embodiment of the present invention.

Referring to FIG. 9 , when CTU encoding starts, the video encoder generates a combined image by combining the input image and the surrounding image based on the encoding time, and uses the combined image and quantization parameters as inputs to generate the current The mode of the video encoder for the CTU is determined (S1).

In step S1, the video encoder generates a combined image by combining the current block corresponding to the input image and neighboring blocks located around the current block. In this case, the neighboring block may be determined using the CTU of the previous frame or the pre-processed CTU.

Then, the video encoder transforms and quantizes the CTU according to the determined mode (S2), and generates a bitstream by performing entropy encoding (S3).

On the other hand, after step S2, the video encoder restores the pixel through inverse quantization and inverse transformation, and puts the reconstructed pixel into a Decoded Picture Buffer (DPB) through deblocking filtering and sample adaptive offset filtering. can be saved In this case, the video encoder may utilize a difference value from a prediction block obtained through motion compensation and intra compensation for predicting motion based on a frame stored in the corresponding DPB as an input for transformation and quantization of the CTU.

The video encoding method using the optimal mode determination according to an embodiment of the present invention may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium.

On the other hand, the embodiments disclosed in the present specification and drawings are merely presented as specific examples to aid understanding, and are not intended to limit the scope of the present invention. It is obvious to those of ordinary skill in the art to which the present invention pertains that other modifications based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. In addition, although specific terms have been used in the present specification and drawings, these are only used in a general sense to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention.

10, 11: Image combining device 20, 21: Mode determining device
30: mode selection device 100, 200: optimal mode determination device

Claims (6)

an image combining device for generating a combined image by combining an input image and a surrounding image based on an encoding time;
A mode determination device including a plurality according to the size of the combined image, and determining the mode of the video encoder for the current Coding Tree Unit (CTU) based on a Convolutional Neural Network (CNN) by inputting the combined image and the quantization parameter as an input ; and
a mode selection device for selecting any one of a plurality of modes output from the plurality of mode determining devices;
and the mode selection apparatus selects a mode having the lowest rate-distortion cost among the plurality of modes output from the plurality of mode determination apparatuses. According to claim 1,
The image combining device combines the current block corresponding to the input image and the neighboring block corresponding to the neighboring image and located in the vicinity of the current block, wherein the neighboring block is determined using a CTU of a previous frame or a pre-processed CTU. An apparatus for determining an optimal mode of a video encoder, characterized in that. delete generating, by a video encoder, a combined image by combining an input image and a surrounding image based on an encoding time;
The video encoder determines the mode of the video encoder for the current Coding Tree Unit (CTU) based on a CNN (Convolutional Neural Network) by inputting the combined image and the quantization parameter, but a plurality of modes according to the size of the combined image determining each;
selecting, by the video encoder, a mode having the lowest rate-distortion cost among the determined plurality of modes;
transforming and quantizing the input CTU according to the selected mode by the video encoder; and
encoding, by the video encoder, transform coefficients obtained through transform and quantization using an entropy coding engine;
A video encoding method using optimal mode determination, comprising: 5. The method of claim 4,
reconstructing a pixel through inverse quantization and inverse transformation after the transformation and quantization;
storing the reconstructed pixels in a decoded picture buffer (DPB) through deblocking filtering and sample adaptive offset filtering;
utilizing, as an input for CTU transformation and quantization, a difference value from a prediction block obtained through motion compensation and intra compensation processes for predicting motion based on the frame stored in the DPB;
Video encoding method using optimal mode determination, characterized in that it further comprises. 5. The method of claim 4,
A video encoding method using optimal mode determination, characterized in that the steps of transforming and quantizing and encoding for a specific CTU are performed independently from the step of determining a mode of a video encoder for another CTU.

Images