TW201924340A - Image encoder using machine learning and data processing method of the image encoder - Google Patents

Abstract

An image encoder for outputting a bitstream by encoding an input image includes a predictive block, a machine learning based prediction enhancement (MLBE) block, and a subtractor. The predictive block is configured to generate a prediction block using data of a previous input block. The MLBE block is configured to transform the prediction block into an enhanced prediction block by applying a machine learning technique to the prediction block. The subtractor is configured to generate a residual block by subtracting pixel data of the enhanced prediction block from pixel data of a current input block.

Description

Image encoder using machine learning and data processing method of image encoder

The disclosure relates to an electronic device. More specifically, the present disclosure relates to an electronic device having a video encoder using machine learning technology, and an encoding method of the image encoder/encoding method for the image encoder.

The demand for high definition video services with high resolution, high frame rate, high bit depth and the like has rapidly increased. Therefore, the importance of codecs for efficiently encoding and decoding a large amount of video data has attracted attention.

H.264 or advanced video coding (AVC) technology involves video compression and provides enhanced performance in terms of compression efficiency, image quality, bit rate, etc. compared to previous video compression techniques. Such video compression technology has been commercialized by digital television (TV) and widely used in various applications, such as video telephony, video conferencing, digital versatile disc (DVD), games and Three-dimensional (3D) TV. H.264 or AVC compression technology currently provides superior performance in terms of compression efficiency, image quality, bit rate, etc. compared to previous versions. However, the motion prediction mode can be more complicated in this technique, and thus the limit of compression efficiency can be gradually reached.

Embodiments of the present disclosure provide an image encoder and an encoding method of the image encoder, the image encoder for generating a prediction block as an enhanced prediction region having a small difference with respect to (as compared to) a source block Block without adding control data.

According to an aspect of an embodiment, a video encoder that outputs a bit stream by encoding an input image includes: a prediction block, a machine learning based prediction enhancement (MLBE) block, and a subtractor. The prediction block is configured to generate a prediction block using data of a previously input block. The machine learning prediction enhancement block is configured to transform the prediction block into an enhanced prediction block by applying machine learning techniques to the prediction block. The subtractor is configured to generate a residual block of residual data by subtracting pixel data of the enhanced prediction block from pixel data of the current input block.

According to another aspect of an embodiment, a method of processing image data includes: generating a prediction block from time domain data of a previously input block; applying at least one of a plurality of available machine learning techniques to the prediction block And converting the prediction block into an enhanced prediction block; and generating a residual block of residual data by subtracting the enhanced prediction block from the current input block.

According to another aspect of an embodiment, a method of processing image data includes: generating a prediction block from time domain data of a previously input block; applying at least one of a plurality of available machine learning techniques to the prediction block And converting the prediction block into an enhanced prediction block; using rate-distortion optimization (RDO) corresponding to each of the prediction block and the enhanced prediction block And selecting one of the prediction block and the enhanced prediction block; and generating a residual block of residual data by subtracting the selected block from the current input block.

In the following, the embodiments of the present disclosure are set forth with reference to the accompanying drawings. Hereinafter, the term "image" in this specification has a comprehensive meaning including moving images (for example, video) and still images (for example, photos).

FIG. 1 is a block diagram showing a configuration of a machine learning (MLB) encoder according to an embodiment of the present disclosure. Referring to FIG. 1, the MLB encoder 100 may divide the input image 10 into a plurality of blocks and may perform MLB predictive encoding on each of the blocks.

The MLB encoder 100 can process the input image 10 to produce an output data 20. The MLB encoder 100 in accordance with embodiments of the present disclosure may use machine learning techniques to generate predicted images and prediction blocks. For example, when the prediction block is generated using the input image 10, the MLB encoder 100 can apply parameters learned using machine learning. That is, the machine learning algorithm can have decision parameters learned using a plurality of predetermined training data sets. In such a case, the prediction block may be close to the source block without increasing the header data 22 when using machine learning. When the prediction block and the source block are closer to each other, the size of the residual data 24 can be reduced by a larger amount.

The output data 20 generated by the MLB encoder 100 applying machine learning techniques in accordance with embodiments of the present disclosure may generally include header data 22 and residual data 24. The MLB encoder 100 according to an embodiment of the present disclosure may perform MLB predictive coding to encode a residual block of a prediction block and a source block. In this case, the data corresponding to the residual block may be the residual data 24. That is, the residual data of the residual block may be the residual data 24. On the other hand, the motion data, the image data, and various set values required for prediction can be output as the header data 22. As the difference between the predicted block and the source block becomes smaller, the size of the residual data 24 can be reduced by a larger amount.

In general, the more accurate the prediction is to reduce the amount of information on the residual blocks and residual images, the more the amount of information needed to predict the header data 22 tends to be more. However, if machine learning in accordance with embodiments of the present disclosure is used, accurate predictions can be achieved without increasing the header data 22. The previously generated prediction block can be enhanced using machine learning to approach the source block. In this case, although the difference between the prediction block and the source block does not have a large influence on the size of the header data 22, the difference between the prediction block and the source block can be effectively reduced. Therefore, the size of the residual data 24 corresponding to the difference between the predicted block and the source is greatly reduced.

FIG. 2 is a block diagram showing a schematic configuration of the MLB encoder shown in FIG. 1. Referring to FIG. 2, the MLB encoder 100 includes a subtractor 110, an MLBE block 120 (MLB prediction enhancement block), a transformer/quantizer 130, a prediction block 140, and an entropy coder 150.

The description herein may refer to structural device/device elements such as encoders, blocks, and encoder devices as representative elements of an encoder (eg, an MLB encoder or a video encoder). Any such representative component can be implemented by a circuit component or a circuit formed by a plurality of circuit components, as appropriate. In addition, any such representative component can be implemented by a processor (eg, a central processing unit, a microcontroller, a microprocessor, or a digital signal processor) that executes a particular set of dedicated software instructions (eg, a software module). The implementation may be implemented by a combination of a processor and a software instruction. Thus, an encoder, block, and encoder device as a structural device/device component can be implemented using circuitry and circuitry, and/or utilizing one or more processors and software executed by the one or more processors The combination of instructions is implemented. Such a processor may execute software instructions to perform one or more processes caused by the elements of interest described herein.

Any of the processors (or similar components) described herein are both tangible and non-transitory. The term "non-transitory" as used herein shall not be interpreted as an eternal character of the state, but rather as a state characteristic that will last for a period of time. The term "non-transitory" specifically denies short-lived characteristics, such as the characteristics of a particular carrier or signal, or other forms that are only temporarily present at any time and any place. The processor is an article of manufacture and/or a machine component. The processor is configured to execute software instructions to perform functions as described in various embodiments herein. The processor can be a general purpose processor or can be part of an application specific integrated circuit (ASIC). The processor can also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. The processor can also be a logic circuit (including, for example, a programmable gate array (PGA) such as a field programmable gate array (FPGA)) or another logic including discrete gate and/or transistor logic. One type of circuit. The processor can be a central processing unit (CPU). Additionally, any of the processors described herein can include multiple processors, parallel processors, or both. The set of instructions can be read from a computer readable medium. Additionally, the instructions, when executed by a processor, can be used to perform one or more of the methods and processes described herein. In a particular embodiment, the instructions may reside entirely or at least partially within the main memory, static memory, and/or processor during execution.

In an alternate embodiment, a dedicated hardware implementation can be implemented to implement one or more of the methods described herein, such as an application-specific integrated circuit (ASIC), which can be programmed. Logic arrays and other hardware components for the functional blocks, bus protectors, and system managers described herein. One or more embodiments described herein may be implemented using two or more dedicated interconnect hardware modules or devices having associated control signals and data signals, the associated control signals and data signals being available in various modes. Transfer between groups and transfer via module. Therefore, the present disclosure encompasses a software implementation, a firmware implementation, and a hardware implementation. Nothing in this application should be construed as being limited to the use of the software only, but not hardware (for example, a tangible non-transitory processor and/or memory).

The subtractor 110 may generate a residual block by the difference between the input block and the generated prediction block. The transformer/quantizer 130 may transform the residual block to output transform coefficients. The transformer/quantizer 130 may quantize the transform coefficients using at least one of a quantization parameter and a quantization matrix. Therefore, quantized coefficients can be produced. In this case, the quantized coefficients output may ultimately correspond to the residual data Residual_Data.

The entropy encoder device 150 may perform entropy encoding using the generated residual data Residual_Data or header data Header_Data (eg, encoding parameters generated during encoding). Entropy coding is a type of lossless coding for compressing digital data by representing frequently occurring patterns with relatively few bits and representing infrequently occurring patterns with relatively many bits. An example of entropy coding is provided below. The bit stream Bitstream can be output by the entropy coding calculation of the entropy encoder device 150. If entropy coding is applied, a small number of bits can be assigned to symbols with high probability of occurrence, and a large number of bits can be assigned to symbols with low probability of occurrence. Therefore, the size of the bit stream of the symbol to be encoded can be reduced according to such symbol representation.

The prediction block 140 may generate the prediction block P based on the input quantized residual data Residual_Data and various parameters. The prediction block 140 may perform encoding in an intra-frame mode or an inter-frame mode to output the prediction block P. The prediction block 140 may generate the prediction block P of the source block S of the input image 10 and may provide the generated prediction block P to the MLBE block 120.

The MLBE block 120 may process the prediction block P to output an enhanced prediction block EP as a processing result. The MLBE block 120 can include a processor that performs an algorithm (eg, one or more available machine learning algorithms) to process the prediction block P and transform the prediction block P into an enhanced prediction block. The MLBE block 120 may use, for example, a machine learning algorithm to process the prediction block P to have a degree close to the source block S to obtain a processing result. In other words, the MLBE block 120 can select the best machine learning techniques among various machine learning techniques with reference to various information (eg, prediction mode, characteristics of motion vectors, partitioned form of the image, size of the transform unit). For example, a decision tree, a neural network (NN), a convolutional neural network (CNN), a support vector machine (SVM), and a K nearest neighbor (K-nearest neighbor, K) can be used. -NN) Various techniques such as algorithm and reinforcement learning are used as machine learning techniques.

In accordance with an embodiment of the present disclosure, the MLB encoder 100 processes the prediction block P to be close to the source block S using machine learning. The disclosure does not require additional information to generate an enhanced prediction block EP. In this context, the enhanced prediction block EP can be provided by the MLBE block 120, which provides the best filtering effect by learning. The MLBE block 120 can maintain or update performance by (online or offline) learning without providing additional material. Therefore, according to the MLB encoder 100 according to the embodiment of the present disclosure, the residual material 24 shown in FIG. 1 can be reduced without increasing the header data 22 of FIG.

Fig. 3 is a block diagram showing a detailed configuration of the MLB encoder shown in Fig. 2. Referring to FIG. 3, the MLB encoder 100 includes a subtractor 110, an MLBE block 120, a transformer 132, a quantizer 134, a prediction block 140, an entropy encoder device 150, and an encoder device controller 160. Herein, the prediction block 140 includes an inverse quantizer 141, an inverse transformer 142, an adder 143, an in-loop filter 144, a buffer 145, a motion estimation block 146, and a motion compensation block. 147. In-frame prediction block 148 and mode decision block 149. The MLB encoder 100 of the above configuration can provide MLB prediction enhancement. Therefore, the residual material 24 shown in Fig. 1 can be reduced without increasing the header data 22 of Fig. 1.

The subtractor 110 may generate a residual block of residual data, which is a difference data between the input block (or source block) and the predicted block. In detail, the subtractor 110 may calculate the value of the current spatial domain block to be currently processed in the plurality of spatial domain blocks included in the input frame and the value of the enhanced prediction block EP output from the MLBE block 120. The difference between. The subtractor 110 may generate a value of a spatial domain residual block corresponding to the calculated difference (hereinafter referred to as "residual data").

For data processing, each of the spatial domain blocks may include m ́n pixels. Herein, each of m and n may be a natural number greater than or equal to 2, and m may be equal to n or m is not equal to n. The pixels included in the spatial domain block may be, but are not limited to, data having a luminance and chrominance (YUV) format, data having a YCbCr format, or having red, green, and blue (RGB) formats. data of. For example, the spatial domain block may include, but is not limited to, 4×4 pixels, 8×8 pixels, 16×16 pixels, 32×32 pixels, or 64×64 pixels. The subtractor 110 may calculate a difference value for each of the calculation blocks and may output the calculated difference value for each of the spatial domain blocks. For example, the size of the calculation block can be smaller than the space domain block. For example, when the calculation block includes 4×4 pixels, the spatial domain block may include 16×16 pixels. However, the size of the calculation block and the size of the spatial domain block are not limited to this.

Transformer 132 may transform the residual block to output transform coefficients. Transformer 132 may perform a time domain to frequency domain transform on the block values included in the spatial domain residual block. For example, transformer 132 can transform the spatial coordinates of the time domain into values of the frequency domain. For example, converter 132 may generate a frequency domain coefficient from the value of the spatial domain residual block using a discrete cosine transform (DCT). In other words, the transformer 132 can transform the residual data as time domain data into frequency domain data.

Quantizer 134 may quantize the input transform coefficients using at least one of a quantization parameter and a quantization matrix. Quantizer 134 may output the quantized coefficients as a result of the quantization. That is, the quantizer can be configured to output the quantized coefficients by quantizing the frequency domain data.

The entropy encoder device 150 may perform entropy encoding to output a bit stream based on a value calculated by the quantizer 134 or an encoding parameter or the like calculated in the encoding process. If entropy coding is applied, a small number of bits can be assigned to symbols with high probability of occurrence, and a large number of bits can be assigned to symbols with low probability of occurrence. Therefore, the size of the bit string of the symbol to be encoded can be reduced. The entropy coder device 150 may use, for example, exponential-Golomb coding, context-adaptive variable length coding (CAVLC), or context-adaptive binary arithmetic coding (context-adaptive binary arithmetic coding). Encoding method such as CABAC) performs entropy coding.

The currently encoded block or image may need to be decoded or stored for use as a reference block or image. Therefore, the coefficients quantized by the quantizer 134 can be inverse quantized by the inverse quantizer 141 and inverse transformed by the inverse transformer 142. The inverse quantized and inverse transformed coefficients may become reconstructed residual blocks and may be added to prediction block P by adder 143. Therefore, a reconstructed block can be generated.

The reconstructed block computed from adder 143 can be transmitted to intra-frame prediction block 148 and can be used to predict intra directional mode. The reconstructed block output from the adder 143 can also be transferred to the in-loop filter 144.

The in-loop filter 144 may apply a deblocking filter, a sample adaptive offset (SAO) filter, or an adaptive loop filter to the reconstructed block or the reconstructed image. At least one of , ALF). The deblocking filter removes block distortion that occurs in the boundary between the various blocks. The SAO filter can add an appropriate offset value to the pixel value to compensate for the coding error. The ALF may perform filtering based on values obtained by comparing the reconstructed block to the source block. The reconstructed block processed by/by the in-loop filter 144 can be stored in the buffer 145 to store the reference image.

The buffer 145 can store the reconstructed block output from the in-loop filter 144 and can provide the reconstructed block to the motion estimation block 146 and the motion compensation block 147. The buffer 145 can provide the reconstructed block output from the in-loop filter 144 as an output data Output data input to the entropy encoder device 150.

In the intra-frame mode, intra-frame prediction block 148 may perform spatial prediction using the pixel values of previously coded blocks around the current block to produce first prediction block P1 as a result of performing spatial prediction. In the inter-frame mode, motion estimation block 146 may find a reference block that most closely matches the input block stored in the reference image in buffer 145 during motion prediction to obtain a motion vector. Motion compensation block 147 may perform motion compensation using motion vectors to generate second prediction block P2. Herein, the motion vector may be a two-dimensional (2D) vector for inter-frame prediction and may represent an offset between a reference block and a block that is currently to be encoded/decoded.

The mode decision block 149 may receive the current block, the first predicted block P1 provided from the motion compensation block 147, and the second predicted block P2 provided by the intra-frame prediction block 148. The mode decision block 149 may determine one of the first prediction block P1 and the second prediction block P2 as the prediction block P and may provide the prediction block P to the MLBE block 120. The mode decision block 149 may determine and output the prediction block P according to the current block value, the inverse quantized coefficient, the block value of the first prediction block P1, and the block value of the second prediction block P2, and the control signal CNT3.

In another embodiment, mode decision block 149 may apply a machine learning algorithm to each of first prediction block P1 and second prediction block P2 and perform mode decision. For example, mode decision block 149 can apply a machine learning algorithm to first prediction block P1 to generate an enhanced first prediction block EP1. The mode decision block 149 may apply a machine learning algorithm to the second prediction block P2 to generate an enhanced second prediction block EP2. The mode decision block 149 may determine one of the enhanced first prediction block EP1 and the enhanced second prediction block EP2 as the prediction block P and may provide the prediction block P to the MLBE block 120. In this case, a value indicating whether processing has been performed by the machine learning algorithm should be included in the enhanced first prediction block EP1 or the enhanced second prediction block EP2. The image compression ratio can be further enhanced by such operation of mode decision block 149.

The MLBE block 120 may process the prediction block P provided from the mode decision block 149 to output the enhanced prediction block EP as a processing result. The MLBE block 120 performing the MLB processing may process the prediction block P to a degree close to the source block S. The MLBE block 120 can select one of various available machine learning techniques with reference to the encoded information MLS_Info. The encoded information MLS_Info may include various information such as a prediction mode previously determined by the mode decision block 149, a feature of the motion vector, a partitioned form of the image, and a size of the transform unit. Various techniques such as decision tree, CNN, SVM, K nearest neighbor (K-NN) algorithm, and reinforcement learning can be used as machine learning techniques. A description will be given in detail of the detailed characteristics of the MLBE block 120 with reference to the drawings.

The encoder device controller 160 can control the overall components in the MLB encoder 100 based on the input image or block. Encoder device controller 160 may determine the partition of the input image, the size of the encoded block, etc. and may control the encoding and decoding of the image in accordance with the determined criteria. The encoder device controller 160 may generate a plurality of control signals CNT1 to CNT4 for such control operations and may provide the control signals CNT1 to CNT4 to the motion estimation block 146, the converter 132, the mode decision block 149, and the inverse quantizer 141, respectively. . The encoder device controller 160 may provide the control data Control data included in the header data 22 (see FIG. 1) of the bit stream to the entropy encoder device 150.

As described above, according to the embodiment of the present disclosure, the MLB encoder 100 processes the prediction block P to be close to the source block S using machine learning. In accordance with an embodiment of the present disclosure, the MLB encoder 100 includes an MLBE block 120 for providing optimal reconstruction effects by learning. The enhanced prediction block EP can be provided by the MLBE block 120 without increasing the residual data. The MLBE block 120 can maintain or update performance by learning (online or offline). Therefore, according to the MLB encoder 100 according to the embodiment of the present disclosure, the residual material 24 can be reduced without increasing the header data 22.

FIG. 4 is a block diagram showing the characteristics of the MLBE block 120 shown in FIG. Referring to FIG. 4, the MLBE block 120 may transform the prediction block P into an optimal enhanced prediction block EP using various encoding information MLS_Info.

The MLBE block 120 can have various machine learning algorithms ML1 through MLn (where n is an integer). The MLBE block 120 can use the encoded information MLS_Info to select a machine learning algorithm with the best enhanced performance. In this context, it should be fully understood that the various machine learning algorithms ML1 through MLn are each provided to a machine learning device that utilizes hardware implementation. In this context, machine learning algorithms ML1 through MLn may include various algorithms, such as decision trees, CNN, SVM, and reinforcement learning.

The coding information MLS_Info may include various parameter conditions, such as a prediction mode, a feature of a motion vector, an in-frame direction, a size of a coding unit, a partition form of an image, and a size of a transform unit. It is well known that machine learning algorithms ML1 to MLn have different filter characteristics for a particular image or feature. Therefore, the quality of the enhanced prediction block EP may vary depending on various conditions or combinations of conditions. The MLBE block 120 in accordance with embodiments of the present disclosure may select an optimal machine learning algorithm determined by the learning process and may generate an enhanced prediction mode EP proximate to the source block S without increasing the header data 22 of FIG. . Therefore, the residual data 24 can be reduced without increasing the header data 22 (see Fig. 1).

5A and 5B are block diagrams showing MLBE blocks for selecting an optimal machine learning algorithm for each prediction mode. FIG. 5A illustrates an MLBE block 120 for selecting a second machine learning algorithm ML2 of a plurality of available machine learning algorithms in an intra-frame mode. FIG. 5B illustrates an MLBE block 120 for selecting a third machine learning algorithm ML3 of a plurality of available machine learning algorithms in an inter-frame mode.

Referring to FIG. 5A, if the encoded information MLS_Info provided to the MLBE block 120 is an intra-frame mode, the intra-frame prediction block 148 is transmitted from the intra-frame prediction block 148 shown in FIG. The intra-frame prediction block P_Intra in the in-frame mode can be generated using only information in a limited screen. Therefore, the intra-frame prediction block P_Intra can be relatively coarse in terms of resolution or quality. The MLBE block 120 may select the second machine learning algorithm ML2 to process such intra-frame prediction block P_Intra as an enhanced prediction block EP having a degree close to the source block S. Such selection can be performed based on the results of various learnings performed previously.

Referring to FIG. 5B, if the encoded information MLS_Info provided to the MLBE block 120 is an inter-frame mode, the inter-frame prediction block P_Inter is transmitted from the motion compensation block 147 shown in FIG. The inter-frame prediction block P_Inter can be generated by referring to another frame previously processed in the image. Therefore, the inter-frame prediction block P_Inter may be relatively finer in resolution than the intra-frame prediction block P_Intra generated in the intra-frame mode, or may have intra-frame prediction generated in the intra-frame mode. Block P_Intra higher / better quality. The MLBE block 120 may select the third machine learning algorithm ML3 to process such inter-frame prediction block P_Inters into an enhanced prediction block EP having a degree close to the source block S. Such selection can be performed based on the results of various learnings performed previously.

As described above, the method for selecting a machine learning algorithm based on a prediction mode is only described as being selected among various independent alternatives. However, this may be merely an exemplary embodiment. It should be fully understood that one or more machine learning algorithms can be combined and applied in various ways according to a combination of various encoding information MLS_Info.

6A and 6B are flowcharts illustrating an encoding method of selecting a machine learning technique according to characteristics of a prediction block, according to an embodiment of the present disclosure. 6A and 6B, an illustration of exemplary operational characteristics of the MLBE block 120 (see FIG. 4) in accordance with an embodiment of the present disclosure will be presented.

Referring to FIG. 6A, MLBE block 120 may select a machine learning algorithm based on the characteristics of the predicted block.

In operation S110, the MLBE block 120 may receive the prediction block P and the encoding information (MLS_Info) shown in FIG. The coded information MLS_Info may include various parameters or conditions, such as a prediction mode, a magnitude or direction of a motion vector, an in-frame direction, a size of a coding unit CU, a partitioned form of an image, and a size of a transform unit. Such encoded information MLS_Info may be provided from mode decision block 149, encoder device controller 160, motion estimation block 146, in-loop filter 144, etc., as shown in FIG. However, it should be fully understood that the type or scope of the encoded information MLS_Info is not limited to this. A combination of the encoded information MLS_Info in various data forms can be used to generate an enhanced prediction block EP with high accuracy.

In operation S120, the MLBE block 120 may check and analyze the provided encoded information MLS_Info. The MLBE block 120 may classify the provided information (eg, the prediction mode, the magnitude or direction of the motion vector, the in-frame direction, the size of the coding unit CU, the partitioned form of the image, and the size of the transform unit) according to predetermined criteria. The predetermined criteria may include information indicating whether any information is applied first and a detailed operational flow based on the corresponding information.

In operation S130, the MLBE block 120 may check the prediction mode and may branch to the operation. To simplify the description in the embodiment, it may be assumed that the MLBE block 120 determines the machine learning technique based on the prediction mode and the motion vector. Of course, it should be fully understood that a combination of various coding information MLS_Info can be applied to determine machine learning techniques. If the prediction mode is the intra-frame mode, the flow may move to operation S180. In contrast, if the prediction mode is the inter-frame mode, the flow may move to operation S140.

In operation S140, the operation may be branched according to the motion vector. To simplify the description, it can be assumed that the operation branches according to the direction of the motion vector. If the motion vector corresponds to the first direction Dir1, the flow may move to operation S150. On the other hand, if the motion vector corresponds to the second direction Dir2, the flow may move to operation S160. If the motion vector corresponds to the third direction Dir2, the flow may move to operation S170.

In each of operations S150 through S180, the prediction block P may be processed according to the selected machine learning technique. As an illustrative example, in operation S150, the prediction block P may be processed according to the decision tree machine learning algorithm ML1. In operation S160, the prediction block P may be processed according to the CNN machine learning algorithm ML2. In operation S170, the prediction block P may be processed according to the SVM machine learning algorithm ML3. In operation S180, the prediction block P may be processed according to a K-nearest neighbor (K-NN) algorithm type machine learning algorithm ML4 suitable for pattern recognition and decision. Additionally, a reinforcement learning algorithm or various machine learning algorithms may be used to generate the prediction block P as an enhanced prediction block EP in accordance with embodiments of the present disclosure.

In operation S190, the MLBE block 120 may output the enhanced prediction block EP generated by the selected machine learning algorithm. The output enhanced prediction block EP will be transmitted to the subtractor 11 (see Fig. 3).

As described above, the type of machine learning algorithm can be selected based on the characteristics of the prediction block. However, the advantages of the present disclosure are not limited to the above embodiments. An explanation of another characteristic will be given below with reference to FIG. 6B.

Referring to FIG. 6B, one of various parameter sets may be selected in a machine learning algorithm (eg, CNN) based on the characteristics of the prediction block. In this paper, CNN is illustrated as an example of a machine learning algorithm. However, it should be fully understood that the disclosure is not limited thereto.

In operation S210, the MLBE block 120 may receive the prediction block P and the encoding information (MLS_Info) shown in FIG. The coded information MLS_Info may include various parameters or conditions, such as a prediction mode, a magnitude or direction of a motion vector, an in-frame direction, a size of a coding unit CU, a partitioned form of an image, and a size of a transform unit. Such encoded information MLS_Info may be provided from mode decision block 149, encoder device controller 160, motion estimation block 146, in-loop filter 144, and the like.

In operation S220, the MLBE block 120 may check and analyze the provided encoded information MLS_Info. The MLBE block 120 may classify the provided information (eg, the prediction mode, the magnitude or direction of the motion vector, the in-frame direction, the size of the coding unit CU, the partitioned form of the image, and the size of the transform unit) according to predetermined criteria. The predetermined criteria may include information indicating whether any information is applied first and a detailed operational flow based on the corresponding information.

In operation S230, the MLBE block 120 may check the prediction mode and may branch to the operation. To simplify the description in the embodiment, it may be assumed that the MLBE block 120 determines the machine learning technique based on the prediction mode and the motion vector. Of course, it should be fully understood that a combination of various coding information MLS_Info can be applied to determine machine learning techniques. If the prediction mode is the intra-frame mode, the flow may move to operation S280. On the other hand, if the prediction mode is the inter-frame mode, the flow may move to operation S240.

In operation S240, the operation may be branched according to the motion vector. To simplify the description, it can be assumed that the operation branches according to the direction of the motion vector. If the motion vector corresponds to the first direction Dir1, the flow may move to operation S250. On the other hand, if the motion vector corresponds to the second direction Dir2, the flow may move to operation S260. If the motion vector corresponds to the third direction Dir3, the flow may move to operation S270.

In each of operations S250 to S280, the prediction block P may be processed according to the selected parameter set. As an illustrative example, in operation S250, the prediction block P may be processed according to a CNN algorithm set to the first parameter set. In operation S260, the prediction block P may be processed according to a CNN algorithm set to the second parameter set. In operation S270, the prediction block P may be processed according to a CNN algorithm set to a third parameter set. In operation S280, the prediction block P may be processed according to a CNN algorithm set to a fourth parameter set. Although the embodiment is illustrated as being divided into four parameter sets, the embodiments of the present disclosure are not limited thereto.

In operation S290, the MLBE block 120 may output the enhanced prediction block EP generated by the machine learning algorithm of the selected parameter set. The output enhanced prediction block EP will be transmitted to the subtractor 11 (see Fig. 3).

As described above, the type of machine learning can be selected by the MLBE block 120 based on the encoded information MLS_Info, and the set of parameters in the same machine learning algorithm can be selected by the MLBE block 120 in accordance with embodiments of the present disclosure. Since the optimal machine learning algorithm or parameter set corresponding to the various prediction blocks P is selected, the difference between the enhanced prediction block EP and the source block S can be minimized.

FIG. 7 is a block diagram showing a training method of an MLBE block according to an embodiment of the present disclosure. Referring to Figure 7, the machine learning algorithms included in the MLBE block 120 can be learned or trained offline using various patterns or images.

Each of the machine learning algorithms ML1 through MLn of the MLBE block 120 can be trained by input of the source block S 121 and the prediction block P 122. For example, in the case of an NN machine learning algorithm, prediction block P 122 may represent a plurality of different prediction blocks of source block S 121. For example, the machine learning parameter ML parameters may be updated such that the prediction block P 122 generated by the various prediction modes is mapped to the source block S 121.

Training using source block S 121 and prediction block P 122 may be performed on each of machine learning algorithms ML1 through MLn. If various previously prepared images or patterns are trained, the parameters of each of the machine learning algorithms ML1 through MLn may be fixed. For example, in an ImageNet scenario as a collection of data for training CNN, approximately 14,000,000 or more than 14,000,000 training images may be used. Thus, each machine learning algorithm can have decision parameters learned using one or more predetermined sets of training data.

If the learning or training process described above is completed, the MLBE block 120 may generate an enhanced prediction block EP having a value most similar to the source block S 121 with respect to the prediction block P 122 input according to the encoding information MLS_Info.

FIG. 8 is a diagram showing a training method of an MLBE block according to another embodiment of the present disclosure. Referring to Figure 8, the machine learning algorithms included in the MLBE block 120 can be trained using images that are processed according to an online training scheme.

In this case, a frame composed of input images can be used to train the machine learning algorithm without using an advanced training machine learning algorithm. If the training session ends, then the training results can be used to perform only parameter updates. For example, the training of the MLBE block 120 can be performed using frames F0 through F4 corresponding to the training period (eg, training interval) in the input frame. If the training period ends, only the updating of the parameters for training using the subsequently input frames F5 to F11 may be performed. Thus, if an input image is provided, each machine learning algorithm can be trained using the input image frame, for example during the training interval.

If an online training program is used, there is no need to have a separate data set for training. Machine learning algorithms can be trained using input images. Therefore, the parameter size can be relatively small. However, if the elements and resources used to support online training are not provided, it may be difficult to accommodate proper performance.

FIG. 9 is a block diagram showing an MLB encoder according to another embodiment of the present disclosure. Referring to FIG. 9, the MLB encoder 200 includes a subtractor 210, an MLBE block 220, a transformer 232, a quantizer 234, a prediction block 240, an entropy encoder device 250, and an encoder device controller 260. In this context, prediction block 240 includes inverse quantizer 241, inverse transformer 242, adder 243, in-loop filter 244, buffer 245, motion estimation block 246, motion compensation block 247, intra-frame prediction block 248, and Mode decision block 249. The MLB encoder 100 can selectively provide enhanced functionality of the MLB prediction block. For example, the MLB encoder 200 can determine whether to use the MLBE block 220 based on a rate distortion optimization (RDO) value indicating coding efficiency. Therefore, a block having a compression efficiency better RDO value can be selected as the selected result between the prediction block and the enhanced prediction block.

Herein, in addition to the MLBE block 220 determining whether to use the MLB prediction enhancement based on the RDO value, the subtractor 210, the transformer 232, the quantizer 234, the prediction block 240, the entropy encoder device 250, and the encoder device controller 260 may be Substantially the same as shown in Figure 3. Therefore, the detailed functions of the subtractor 210, the converter 232, the quantizer 234, the prediction block 240, the entropy encoder device 250, and the encoder device controller 260 will be omitted.

On the other hand, the MLBE block 220 may have the enhancement function of the MLB prediction block P shown in FIG. 3 and may additionally determine whether to generate the enhanced prediction block EP or the predicted block P provided by bypass. If it is determined from the RDO value that there is no performance gain caused by processing the prediction block in the machine learning technique, the MLBE block 220 may bypass the prediction block P provided from the mode decision block 249 to the subtractor 210. If it is determined from the RDO value that there is a performance gain caused by processing the prediction block in the machine learning technique, the MLBE block 220 may process the prediction block P provided from the mode decision block 249 using machine learning techniques and may enhance The prediction block (EP) is transmitted to the subtractor 210 as a result of the processing.

Overhead caused by prediction enhancement can be prevented by the selective prediction enhancement operation on MLBE block 220 described above. Herein, the RDO value is illustrated as an example of information for judging whether to apply the prediction enhancement operation of the MLBE block 220. However, the MLBE block 220 in accordance with embodiments of the present disclosure may perform selective prediction enhancement operations using various performance parameters and RDO values.

Figure 10 is a block diagram showing the function of the MLBE block shown in Figure 9. Referring to FIG. 10, the MLBE block 220 includes an MLBE block 222 and a selection block 224. The MLBE block 222 can transform the prediction block P into the optimal enhanced prediction block EP using various encoding information MLS_Info. Selection block 224 may select one of prediction block P and enhanced prediction block EP.

The MLBE block 222 can have various machine learning algorithms ML1 through MLn. The MLBE block 222 can use the encoded information MLS_Info to select a machine learning algorithm with the best enhanced performance. The MLBE block 222 can perform substantially the same functions as the MLBE block 120 shown in FIG. Therefore, the description of the MLBE block 222 will be omitted.

Selection block 224 may select one of prediction block P and enhanced prediction block EP with reference to the RDO value. The selected one block can be output as the selection prediction block SP supplied to the subtractor 210 shown in FIG. The overhead incurred by applying machine learning can be reduced by selecting the prediction block P or the enhanced prediction block EP according to the RDO value.

Figure 11 is a flow chart showing the operation of the MLBE block shown in Figure 10. Referring to FIG. 11, the MLBE block 220 shown in FIG. 9 can bypass the prediction block P supplied from the mode decision block 249 shown in FIG. 9 to the subtractor 210 shown in FIG. 9 without applying the enhancement operation with reference to the RDO value.

In operation S310, the MLBE block 220 may receive the prediction block P generated from the mode decision block 249.

In operation S320, the MLBE block 220 may calculate an RDO value. The MLBE block 220 may determine whether to perform the MLB prediction enhancement operation or whether to transmit the prediction block P supplied from the mode decision block 249 to the subtractor 210 without performing the MLB prediction enhancement operation according to the RDO value.

In operation S330, if the performance when the prediction block P is processed using machine learning according to the RDO value is equal to less than (sparse) the performance when the machine learning is not used (ML £ non-ML), the flow may move to operation S340. . On the contrary, if the performance when using the machine learning to process the prediction block P is greater than (better than) the performance when the machine learning is not used (ML>non-ML), the flow may move to operation S350.

In operation S340, the MLBE block 220 may select the prediction block P provided from the mode decision block 249 and may transmit the selected prediction block P to the subtractor 210.

In operation S350, the MLBE block 220 may process the prediction block P provided from the mode decision block 249 by the MLBE block 222 to obtain a processing result. The MLBE block 220 may select the enhanced prediction block EP and may transmit the selected enhanced prediction block EP to the subtractor 210.

In operation S360, the MLB encoder 200 may write a flag indicating whether to compress the transmitted bit stream by applying machine learning in the video stream syntax.

As described above, the RDO value can be used to determine whether to apply an activation operation of the MLB prediction block according to the disclosed embodiment. In some cases, a special case may arise where the overhead is increased when the application MLB is enabled. In this case, the MLB encoder 200 according to an embodiment of the present disclosure may select a prediction block P that does not apply machine learning to prevent overhead caused by performing machine learning.

FIG. 12 is a diagram showing an example of a video stream syntax according to an embodiment of the present disclosure explained with reference to FIG. Referring to FIG. 12, it can be confirmed that the MLB prediction enhancement operation according to the embodiment of the present disclosure is applied by representing the syntax of the coding unit.

When the bit stream of the image or block is transmitted, if the MLB prediction enhancement is applied, the MLB encoder 200 shown in Fig. 1 can write '1' in the flag of the video stream syntax (ml_based_pred_enhancement_flag). In contrast, when transmitting a bit stream of an image or a block, if the MLB prediction enhancement is not applied, the MLB encoder 200 can write '0' in the flag of the video stream syntax.

Referring to the flag of the video stream syntax (ml_based_pred_enhancement_flag), the decoder can apply or skip the prediction enhancement operation by machine learning when performing the decoding operation.

Figure 13 is a block diagram showing an MLB decoder. Referring to Figure 13, an MLB decoder 300 in accordance with an embodiment of the present disclosure includes an MLBE block 390 for decoding.

The MLBE block 390 can perform the same or similar operations on the MLBE block 120 shown in FIG. 3 or the MLBE block 220 shown in FIG. In other words, MLBE block 390 can use the encoding information MLS_Info to select the best machine learning algorithm and can use the selected machine learning algorithm to generate prediction block P as the enhanced prediction block EP. Alternatively, MLBE block 390 can select prediction block P or enhanced prediction block EP with reference to a flag included in the video stream syntax. MLB decoder 300 may reconstruct bit stream 30 into output image 40 using such MLB enhanced prediction block EP.

Fig. 14 is a block diagram showing a detailed configuration of the MLB decoder shown in Fig. 13. Referring to FIG. 14, the MLB decoder 300 includes an entropy decoder 310, an inverse quantizer 320, an inverse transformer 330, an adder 340, an in-loop filter 350, a buffer 360, an intra-frame prediction block 370, a motion compensation block 372, Motion estimation block 374, mode decision block 380, and MLBE block 390.

The MLB decoder 300 can receive the bit stream output from the MLB encoder 100 shown in FIG. 1 or the MLB encoder 200 shown in FIG. 9 and can perform decoding in an intra-frame mode or an inter-frame mode to output a reconstructed image. The MLB decoder 300 may obtain the reconstructed residual block from the received bitstream and may generate the predicted block P. If the MLB processing of the prediction block P is performed using the MLBE block 390, an enhanced prediction block EP can be generated. The MLB decoder 300 may add the reconstructed residual block to the enhanced prediction block EP to produce a reconfigured reconstructed block.

The overall elements of MLB decoder 300 may be substantially identical to the overall elements of MLB encoder 100 or 200 described above. Therefore, a detailed description of the elements of the MLB decoder 300 will be omitted below.

FIG. 15 is a block diagram showing a portable terminal for performing an MLB prediction enhancement operation according to an embodiment of the present disclosure. Referring to FIG. 15, a portable terminal 1000 according to an embodiment of the present disclosure includes an image processing unit 1100, a wireless transceiver, an audio processing unit, a power management integrated circuit (PMIC) 1400, a battery 1450, and a memory 1500. User interface 1600 and controller 1700.

The image processing unit 1100 includes a lens 1110, an image sensor 1120, an image processor 1130, and a display unit 1140. The wireless transceiver includes an antenna 1210, a transceiver 1220, and a data machine 1230. The audio processing unit includes an audio processor 1310, a microphone 1320, and a speaker 1330.

Specifically, the image processing unit 1100 according to the embodiment of the present disclosure can process the prediction block by applying machine learning technology. In this case, the image processing unit 1100 can reduce the residual data without increasing the size of the header data of the video signal.

According to the encoder and the encoding method according to the embodiments of the present disclosure, an encoder and decoding with low image quality degradation while enhancing the data compression rate by minimizing the difference between the prediction block and the source block can be implemented. Device.

While the present invention has been described with reference to the exemplary embodiments thereof, it will be apparent to those skilled in the art that various changes and modifications can be made without departing from the spirit and scope of the disclosure. Therefore, it should be understood that the above-described embodiments are not limiting, but illustrative.

10‧‧‧ Input image

20‧‧‧Output data

22‧‧‧ Header information

24‧‧‧Residual data

30‧‧‧ bit stream

40‧‧‧ Output image

100, 200‧‧‧MLB encoder

110, 210‧‧‧ subtractor

120, 220, 222, 390‧‧‧MLBE blocks

121‧‧‧Source Block S

122‧‧‧ Forecast Block P

130‧‧‧Transformer/Quantizer

132, 232‧‧ ‧ inverter

134, 234‧‧ ‧ quantizer

140, 240‧‧‧ forecast block

141, 241, 320‧‧‧ inverse quantizer

142, 242, 330‧‧‧ inverse transformer

143, 243, 340‧‧ ‧ adders

144, 244, 350‧‧‧ In-loop filter

145, 245, 360‧ ‧ buffer

146, 374‧‧ ‧ motion estimation block

147, 247, 372 ‧ ‧ motion compensation block

148, 248, 370‧‧‧ In-frame prediction block

149, 249, 380‧‧‧ mode decision block

150, 250‧‧‧ Entropy encoder device

160, 260‧‧‧Encoder device controller

224‧‧‧Select block

246‧‧‧Sports estimation

300‧‧‧MLB decoder

310‧‧‧ Entropy decoder

1000‧‧‧Portable Terminal

1100‧‧‧Image Processing Unit

1110‧‧‧ lens

1120‧‧‧Image Sensor

1130‧‧‧Image Processor

1140‧‧‧Display unit

1210‧‧‧Antenna

1220‧‧‧ transceiver

1230‧‧‧Data machine

1310‧‧‧Optical processor

1320‧‧‧Microphone

1330‧‧‧Speakers

1400‧‧‧Power Management Integrated Circuit

1450‧‧‧Battery

1500‧‧‧ memory

1600‧‧‧User interface

1700‧‧‧ controller

Bitstream‧‧‧ bitstream

CNT1, CNT2, CNT3, CNT4‧‧‧ control signals

Dir1‧‧‧ first direction

Dir2‧‧‧ second direction

Dir3‧‧‧ third direction

EP‧‧‧ Enhanced Prediction Block/MLB Enhanced Prediction Block

F0, F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, F11‧‧‧ frames

ML1‧‧‧ machine learning algorithm/decision tree machine learning algorithm

ML2‧‧‧Second Machine Learning Algorithm/CNN Machine Learning Algorithm/Machine Learning Algorithm

ML3‧‧‧ third machine learning algorithm/SVM machine learning algorithm/machine learning algorithm

MLn‧‧‧ machine learning algorithm

MLS_Info‧‧‧ Coded Information

P‧‧‧ Forecast Block/MLB Forecast Block

P_Inter‧‧‧ Inter-frame prediction block

P_Intra‧‧‧ In-frame prediction block

P1‧‧‧ first forecast block

P2‧‧‧ second forecast block

S‧‧‧ source block

S110, S120, S130, S140, S150, S160, S170, S180, S190, S210, S220, S230, S240, S250, S260, S270, S280, S290, S310, S320, S330, S340, S350, S360‧‧ operating

SP‧‧‧Select prediction block

The above and other objects and features will be apparent from the following description, wherein the same reference numerals refer to the same. A block diagram showing a configuration of an MLB encoder according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing a schematic configuration of the MLB encoder shown in FIG. 1. Fig. 3 is a block diagram showing a detailed configuration of the MLB encoder shown in Fig. 2. 4 is a block diagram showing the characteristics of the Machine Learning Predictive Enhancement (MLBE) block shown in FIG. 5A and 5B are block diagrams showing MLBE blocks for selecting an optimal machine learning algorithm for each prediction mode. 6A and 6B are flowcharts illustrating an encoding method of selecting a machine learning technique according to characteristics of a prediction block, according to an embodiment of the present disclosure. FIG. 7 is a block diagram showing a training method of an MLBE block according to an embodiment of the present disclosure. FIG. 8 is a diagram showing a training method of an MLBE block according to another embodiment of the present disclosure. FIG. 9 is a block diagram showing an MLB encoder according to another embodiment of the present disclosure. Figure 10 is a block diagram showing the function of the MLBE block shown in Figure 9. Figure 11 is a flow chart showing the operation of the MLBE block shown in Figure 10. FIG. 12 is a diagram showing an example of a video stream syntax according to an embodiment of the present disclosure explained with reference to FIG. Figure 13 is a block diagram showing an MLB decoder. Fig. 14 is a block diagram showing a detailed configuration of the MLB decoder shown in Fig. 13. FIG. 15 is a block diagram showing a portable terminal for performing an MLB prediction enhancement operation according to an embodiment of the present disclosure.

Claims (10)

An image encoder for outputting a stream of bits by encoding an input image, the image encoder comprising: a prediction block configured to generate a prediction block using data of a previously input block; machine learning prediction enhancement (MLBE) a block configured to transform the prediction block into an enhanced prediction block by applying machine learning techniques to the prediction block; and a subtractor configured to input a pixel from the current block The data is subtracted from the pixel data of the enhanced prediction block to generate a residual block of residual data. The image encoder of claim 1, wherein the machine learning prediction enhancement block is configured to: execute a plurality of machine learning algorithms to process the prediction block. The image encoder of claim 2, wherein the machine learning prediction enhancement block is configured to: select at least one of the plurality of machine learning algorithms as a reference with reference to encoding information of the input image Selecting a machine learning algorithm; and processing the predicted block using the selected machine learning algorithm. The image encoder of claim 3, wherein the encoding information comprises at least one of: a prediction mode corresponding to the prediction block, a magnitude and a direction of the motion vector, and an in-frame direction , the size of the coding unit, the partition from the image, and the size of the transform unit. The image encoder of claim 2, wherein the plurality of machine learning algorithms comprise at least one of: a decision tree, a neural network (NN), a convolutional neural network (CNN), a support vector Machine (SVM), reinforcement learning and K nearest neighbor (K-NN) algorithm. The image encoder of claim 1, wherein the machine learning prediction enhancement block is configured to: according to a rate distortion of each of the prediction block and the enhanced prediction block Depending on the RDO value, one of the prediction block and the enhanced prediction block is transmitted to the subtractor. The image encoder of claim 6, wherein the machine learning prediction enhancement block comprises: a machine learning prediction enhancement block configured to select one of a plurality of machine learning algorithms as the selected machine learning calculus a method of processing the prediction block using the selected machine learning algorithm to obtain a processing result, and generating the enhanced prediction block according to the processing result; and selecting a block configured to be based on the prediction And selecting a rate distortion optimization value for each of the block and the enhanced prediction block, and selecting one of the prediction block and the enhanced prediction block as the selected block, and The selected block is transferred to the subtractor. The image encoder of claim 1, wherein the machine learning prediction enhancement block is configured to: execute a machine learning algorithm that selects one of a plurality of parameter sets as selected according to the encoding information The set of parameters is used and the predicted block is processed using the selected set of parameters. A method of processing image data, the method comprising: generating a prediction block from time domain data of a previously input block; and applying the prediction block by applying at least one of a plurality of machine learning techniques to the prediction block Transforming into an enhanced prediction block; and generating a residual block by subtracting the enhanced prediction block from the current input block. A method of processing image data, the method comprising: generating a prediction block from time domain data of a previously input block; and applying the prediction block by applying at least one of a plurality of machine learning techniques to the prediction block Transforming into an enhanced prediction block; selecting the prediction block and the enhancement using a rate distortion optimization (RDO) value corresponding to each of the prediction block and the enhanced prediction block One of the prediction blocks is selected as the block; and the residual block is generated by subtracting the selected block from the current input block.