KR20180100976A - Method and apparatus for picture encoding/decoding using deep neural network based blur image learning - Google Patents

Abstract

Disclosed are a method and an apparatus which can increase compression efficiency by encoding a still image and/or a moving image after performing blur-preprocessing for each block included in the image based on a learning result using a deep neural network, instead of directly encoding the image, and restoring blur images at the time of decoding. An image encoding method according to an embodiment of the present disclosure includes the following steps: dividing an image to be encoded into one or more unit areas; deriving a restoration parameter or blur level for each of the unit areas; performing blurring for each of the unit areas to obtain blur images; and encoding the blurred image. The step of deriving a restoration parameter or blur level for each of the unit areas may be performed by using a model generated through a process of determining one or more sample images and restoration parameters for each of the blurred images of the sample images. The blur level for the unit area may be determined based on the complexity of the unit area.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and apparatus for image encoding / decoding using a deep neural network-based blurred image learning,

The present disclosure is intended to increase the compression efficiency by inputting a stepwise blurred image rather than an original image in still image and / or moving image compression. More specifically, the step blur images of the sample image and the sample images are subjected to learning using a deep neural network, the blurred image is encoded by applying the blurred image to the actual encoding target image, and the compressed image is decoded To a method and apparatus for recovering an original image by applying a blur reconstruction learning result to the blur image.

A conventional compression algorithm for compressing a still image and / or a moving image is basically a method of dividing an image into blocks and applying various algorithms to remove redundant data of the image in the block.

The redundant data in these images is based on redundancy and visibility. Unnecessary redundancy is to remove spatial and / or temporal redundancy information that occurs when the same information is transmitted more than once. Visibility is based on the fact that human eyes are insensitive to color changes, and color information can be transmitted at a lower resolution than luminance (brightness) information.

Image compression is basically the act of analyzing the input original image and reducing redundant information as much as possible, and restoring it to restore it as close as possible to the original.

As described above, a general still / moving image video compression technique divides an image into blocks first, reduces redundant information by using various prediction information in units of blocks, and reduces information beyond a human perception through a conversion technique We use the compression method through the quantization technique.

However, these compression methods basically use a method of recovering original information after loss of the original information, and there is a limitation in completely restoring lost information once.

On the other hand, applying pre-processing and / or post-processing to an image before encoding an image may have a large effect on image compression in order to achieve the above-described description. If the original image can be input and encoded by blurring through the preprocessor before the original image is input to the encoder, and the blurred image can be reconstructed using the post processor after decoding, the encoding efficiency can be increased. However, restoring original image from blur image is very difficult in image processing field. In addition, even if a function of restoring the blur image to an original image by conversion can be obtained, it is highly likely to operate only on the corresponding image. Therefore, if the transform function is directly applied to another image, the blur reconstruction may not operate properly.

Accordingly, there is a need for a method capable of effectively and effectively encoding and decoding an image using a blurred image.

SUMMARY OF THE INVENTION The present invention provides a method and apparatus for restoring an original image in a blurred image by learning a relationship between an original image and blurred images using a deep neural network.

It is another object of the present invention to provide a method and apparatus for reducing a compression capacity by inputting a step-blurred image for each block, rather than coding an original image for still image and / or moving image coding.

According to another aspect of the present invention, there is provided a method for encoding and decoding a blurred image, which is capable of restoring an original image in a blurred image using a learned result of a deep neural network, A method and an apparatus.

The technical objects to be achieved by the present disclosure are not limited to the above-mentioned technical subjects, and other technical subjects which are not mentioned are to be clearly understood from the following description to those skilled in the art It will be possible.

According to one aspect of the present disclosure, a video encoding method can be provided. The method includes dividing an image to be encoded into one or more unit areas, deriving a restoration parameter or blur level for each unit area, blurring each of the unit areas to obtain a blur image And encoding the blurred image.

In the image encoding method according to the present disclosure, the step of deriving the reconstruction parameter or the blur level for each of the unit areas may include determining a reconstruction parameter for each of the one or more sample images and the blurred images of the sample images, Lt; RTI ID = 0.0 > model.

In the image encoding method according to the present disclosure, the blur level for the unit area may be determined based on the complexity of the unit area.

The features briefly summarized above for this disclosure are only exemplary aspects of the detailed description of the disclosure which follow, and are not intended to limit the scope of the disclosure.

According to the present disclosure, blur preprocessing is performed for each block included in an image based on the result of learning using a deep neural network without directly coding still images and / or moving images, and then the blur preprocessing is performed. A method and an apparatus capable of increasing the compression efficiency can be provided.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below will be.

FIG. 1 is a view for explaining the configuration and operation of an image decoder / decoder according to an embodiment of the present disclosure.
FIG. 2 is a diagram for explaining the configuration and operation of the deep neural network-based image learning module 110 according to an embodiment of the present disclosure.
FIG. 3 is a diagram for explaining the configuration and operation of a deep neural network-based image preprocessing module 120 according to an embodiment of the present disclosure.
4 is a diagram for explaining a process of learning reconstruction parameters by inputting blurred images obtained by blurring an original image and an original image according to an embodiment of the present disclosure.
5 is a diagram showing an example of adaptively applying blur steps according to the complexity of an image.
6 is a diagram for explaining an image change according to a deep neural network based image encoding operation according to the present disclosure.
7 is a view for explaining an image change according to a deep neural network based image decoding operation according to the present disclosure.
FIG. 8 is a diagram for explaining a de-neural network-based image encoding process according to an embodiment of the present disclosure.
FIG. 9 is a diagram for explaining a DIP-based image decoding process according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, which will be easily understood by those skilled in the art. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear. Parts not related to the description of the present disclosure in the drawings are omitted, and like parts are denoted by similar reference numerals.

In the present disclosure, when an element is referred to as being "connected", "coupled", or "connected" to another element, it is understood that not only a direct connection relationship but also an indirect connection relationship May also be included. Also, when an element is referred to as " comprising "or" having "another element, it is meant to include not only excluding another element but also another element .

In the present disclosure, the terms first, second, etc. are used only for the purpose of distinguishing one element from another, and do not limit the order or importance of elements, etc. unless specifically stated otherwise. Thus, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly a second component in one embodiment may be referred to as a first component .

In the present disclosure, the components that are distinguished from each other are intended to clearly illustrate each feature and do not necessarily mean that components are separate. That is, a plurality of components may be integrated into one hardware or software unit, or a single component may be distributed into a plurality of hardware or software units. Thus, unless otherwise noted, such integrated or distributed embodiments are also included within the scope of this disclosure.

In the present disclosure, the components described in the various embodiments are not necessarily essential components, and some may be optional components. Thus, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present disclosure. Also, embodiments that include other elements in addition to the elements described in the various embodiments are also included in the scope of the present disclosure.

The emergence of deep learning in the field of artificial intelligence has made great success in the field of machine learning. Basically, deep learning is a type of artificial intelligence developed from artificial neural networks. It learns data by using information input / output hierarchy similar to brain neurons, and can be implemented through a neural network composed of multiple layers of neural networks.

Deep learning learns a very large amount of data, and when new data is input, it chooses the highest answer with probability based on the learning result, so it can adaptively operate even if the environment of the image changes.

In the present disclosure, a tremendous amount of sample images and step-by-step blur images of the images can be learned using the deep neural network technique, and restoration parameters between the original image and the blur images can be obtained. Instead of inputting the original image, a blurred image may be input to the encoder and a restoration parameter may be transmitted along with the encoded image to obtain high compression efficiency. When restoring an image at a decoder, an original image can be restored by applying restoration parameters to the decoded image again.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An apparatus and an operation method constituting the present disclosure with reference to the accompanying drawings will be described in detail with reference to the embodiments of the present disclosure. The following description refers to a specific block module, which is provided for a more general understanding of the present disclosure, and variations and modifications may be made within the scope of the present disclosure.

According to another aspect of the present invention, there is provided a method of decoding an image, the method comprising: obtaining image restoration parameters and / or blur levels by learning sample images; inputting an original image, which is an object to be encoded, A step of encoding the blurred image, a step of decoding the encoded image, and inputting the decoded image into the learned neural network, together with the image restoration parameters obtained in the previous step, to restore the original image.

FIG. 1 is a view for explaining the configuration and operation of an image decoder / decoder according to an embodiment of the present disclosure.

The image encoding / decoding system according to an embodiment of the present disclosure includes a deep neural network based image learning module 110, a deep neural network based image preprocessing module 120, an image encoding module 130, an image decoding module 140 and / And a deep neural network based image post-processing module 150.

FIG. 2 is a diagram for explaining the configuration and operation of the deep neural network-based image learning module 110 according to an embodiment of the present disclosure.

The deep neural network based image learning module 110 may perform a function of learning sample images. 2, the deep nerve-based image learning module 110 includes a sample image DB unit 210, an image divider 220, a video block blur unit 230, an image blur reconstruction unit 240, And / or a blur level determination unit (250). Also, the deep neural network based image learning module 110 may be implemented as a combination of a deep neural network model that performs a plurality of functions.

The sample image DB unit 210 may be a storage device for a sample image DB for learning before actually performing the preprocessing. The sample image DB unit 210 may include images of various sizes and scenes. The images of the sample image DB unit 210 may be tagged according to image characteristics and / or categories for convenience of learning.

The image divider 220 may perform a function of dividing an image into a unit of an appropriate patch to input the image for learning into the deep neural network to learn it in the deep neural network. The size of the patch may vary depending on the level of the hardware and / or software layer network of the deep neural network and / or the characteristics of the image, e.g., the total size of the image.

The image block blur unit 230 may perform image blur on the divided image blocks. The blur level for performing image blur can be adjusted stepwise. For example, the blur level of the first-performed image blur can be set to the lowest level (e.g., blur level 1).

The image blur reconstruction unit 240 can restore the blurred image close to the original image through the inverse process. In the process of restoring the blurred image to the original image, a restoration parameter for the blur level can be derived. For example, it is possible to perform the function of obtaining the restoration parameters by inputting the step blur images in which the step blur is applied to the original sample images and the sample images, analyzing the blur images, and performing learning.

The operations of the image block blurring unit 230 and the image blurring restoring unit 240 may be performed a plurality of times while adjusting the blur level. For example, after performing image blur and image restoration for blur level 1, the blur level may be readjusted to repeat the operation. For example, image blur and image restoration for blur level 1 may be performed, and image blur and image restoration may be performed for blur level 2. [ The above operation can be repeatedly performed from a blur level 1 to a predetermined blur level n (n is an integer of 2 or more).

When image blur and image reconstruction are performed while adjusting the blur level from blur level 1 to blur level n, the image blur reconstruction unit 240 does not directly perform image reconstruction for the blur level n image, , And the blur level n-1 image.

The blur level determination unit 250 can determine an optimal blur level that can be restored to the original image based on the result. For example, by analyzing an image input by a patch unit, it is learned which optimum level of restoration is possible when a blur of a certain level (level) is selected according to complexity, and after learning, a function of determining a blur level by using the result is performed can do. Once the optimal blur level is determined, a restoration parameter for that blur level can be determined.

FIG. 3 is a diagram for explaining the configuration and operation of a deep neural network-based image preprocessing module 120 according to an embodiment of the present disclosure.

The deep neural network based image preprocessing module 120 may use a deep neural network model that has been learned. 3, the deep neural network based image preprocessing module 120 includes an image segmentation unit 310, a deep nerve-based learning model 320, a blur level determination unit 330, and / or an image blur unit 340 ). ≪ / RTI >

The deep neural network-based image preprocessing module 120 can receive an encoding target image, that is, an original image. The image divider 310 can divide the inputted encoding target image into patch units having a predetermined size. The image divider 310 can divide the input image based on the characteristics of the image.

The divided image may be input to the deep neural network-based learning model 320 and / or the blur level determination unit 330. The deep neural network based learning model 320 may include a plurality of models learned by the deep neural network based learning module described with reference to FIG.

The blur level determination unit 330 can determine the blur level of the input divided image using the deep neural network based learning model 320. [ The results of the above-described deep neural network based learning can be used to determine the blur level. The blur level may be determined for each input divided image (e.g., patch) to determine an optimal blur level for each divided image. Once the optimal blur level is determined, the image restoration parameters can be determined accordingly, and in this case also the deep neural network based learning results can be used.

The image blur unit 340 may blur each divided image according to the determined blur level to output a blur image for each patch. At this time, information on blur level and / or information on image restoration parameters may be output together.

4 is a diagram for explaining a process of learning reconstruction parameters by inputting blurred images obtained by blurring an original image and an original image according to an embodiment of the present disclosure.

In FIG. 4, X denotes an original image. The original image X can be input to n deep neural networks (convolution n) having blur levels of 1 to n, respectively. That is, in FIG. 4, convolution n denotes a deep neural network for blurring the original image X at blur level n. A weighted restoration parameter can be derived between images with different blur levels. For example, as described above, to restore the blur level n image, a weight with the blur level n-1 image may be used, and in FIG. 4, W means this weight. Each deep neural network (convolution n) for blurring an image can derive the weighted restoration parameter Wn with another deep neural network.

Images g1 (X) to gn (X) blurred at respective blur levels can be output from respective deep neural networks (convolution n), and gn (X) is a blur image obtained by blurring the original image X at level n .

Images blurred at each blur level can be input to Recon. Recon means a deep neural network for reconstructing an image blurred at each level. The images g1 (X) to gn (X) blurred at respective blur levels can be restored in Recon and output as D (x1) to D (xn). D (xn) denotes an image obtained by reconstructing an image blurred at level n from Recon.

After each level is blurred, each reconstructed image reconstructed in Recon can be input as L. L may play a role in determining an optimized blur level of reconstructed images.

The deep neural network based image learning module 110 receives the blur images g ₁ (X), g ₂ (X) ... g _n (X) blurring the original image X and the original image step by step , Restoration parameters for reconstruction of the original image and blur images can be obtained. Even if a new image is input through the learning process, restoration parameters can be obtained adaptively to the image.

5 is a diagram showing an example of adaptively applying blur steps according to the complexity of an image.

A single blur step can be applied to the entire image when determining the blur step of the image. In some cases, when a single blur step is applied to the entire image, the accuracy of reconstruction may vary depending on the characteristics of the image. For example, when a single blur function is applied to an entire image, a simple portion of a scene can be easily restored, but noise in a portion where a scene of the image is complicated may occur. In order to cope with such a case, according to one embodiment of the present disclosure, it may be desirable to learn, in addition to the restoration parameters at the time of image learning, a certain blur step according to the complexity of the image so as to determine whether optimal restoration is possible. According to the embodiment in which the blur step is adaptively applied according to the complexity of the image, the blur step can be adaptively determined in blocks or patch units of the image upon input of the actual image to be encoded, May be possible.

As shown in FIG. 5, the input image may be divided into a plurality of unit areas. Based on the complexity of the image for each unit area, the blur level for each unit area can be determined. For example, a high blur level (blur level 4) can be applied to the unit areas 510 to 530 and 570 to 590 having low image complexity. Further, an intermediate blur level (blur level 2) can be applied to the unit areas 540 and 560 in which the image complexity is intermediate. Further, a low blur level (blur level 1) can be applied to the unit area 550 having a high image complexity.

As shown in FIG. 5, when the blur level is adaptively applied according to the complexity of each unit area constituting an image, optimal restoration can be performed. However, if the complexity of the configuration is to be considered, a single blur level may be applied to the entire image.

Once the learning of the sample images is completed and the deep neural network based image learning module is constructed, the image for the actual encoding can be inputted. At this time, the image may be a still image and / or a moving image (video).

6 is a diagram for explaining an image change according to a deep neural network based image encoding operation according to the present disclosure.

7 is a view for explaining an image change according to a deep neural network based image decoding operation according to the present disclosure.

6 and 7 may correspond to an image input to each module of the encoder and decoder and an image output from each module. As described above, the blur level can be applied adaptively according to the complexity of the image. However, in the example shown in Figs. 6 and 7, a single blur step is applied to one image for convenience of explanation.

1, 6, and 7, the blur image 620 output from the deep neural network-based image preprocessing module 120 with the original image 610 as input is subjected to image encoding Module 130 as shown in FIG. The image encoding module 130 may compress the blur image 620 and output the encoded image 630. [ Compression of blur image 620 may have higher efficiency than compression of original image 610 directly. The image encoding module 130 may output a restoration parameter for restoring an image together with the encoded image 630 obtained by compressing the blur image 620. [ The coded image 630 and the restoration parameter may be included in the bitstream and transmitted to the image decoding module 140. [

The image decoding module 140 may perform image decoding on the input encoded image 710 and output the decoded image 720 to reproduce the image. If the decoded image output from the image decoding module 140 is an image decoded using a general image codec, the image can be directly rendered through the output device. If the image decoded by the image decoding module 140 is a blurred image coded according to the present disclosure, the image decoding module 140 can output the blurred decoded image 720. [ When the image decoding module 140 outputs the blurred decoded image 720, a process of post-processing the image through the deep neural network based image post-processing module 150 to generate the blurred restoration image 730 is further performed .

In another embodiment of the operation of the de-neural network based image post-processing module 150, the decoded image 720 may have different blur levels for each patch depending on the complexity of the image as described above. In this case, the decoded image 720 can perform a function of restoring each patch into an original image by processing the image using restoration parameters input together for each patch.

According to one embodiment of the present disclosure, since the blurred reconstruction image 730 approximate to the original image 740 can be efficiently reconstructed, it is possible to improve the compression efficiency and generate a reconstructed image of high quality.

FIG. 8 is a diagram for explaining a de-neural network-based image encoding process according to an embodiment of the present disclosure.

In step S810, in order to construct a deep neural network-based image encoding / decoding system, an original image is reconstructed from blur images and an image restoration parameter of sample images for selecting the best blur level according to the complexity of the image and / Learning about the selection can be performed.

After the learning is completed, in step S820, the actual encoding target image can be input. In step S830, the input image may be divided into blocks or patch units (unit division areas) and input. In step S840, the blur level can be adaptively selected for each unit division area according to the complexity of the unit division area. In step S850, a restoration parameter for the blur level adaptively selected may be obtained. In step S860, image blurring for each unit division area may be performed. In step S870, an image to which the blur is applied for each unit division region may be input to an encoder and coded.

FIG. 9 is a diagram for explaining a DIP-based image decoding process according to an embodiment of the present disclosure.

In step S910, the bitstream generated through the encoding process described with reference to FIG. 8 may be input. In step S920, image decoding may be performed. The compressed image is input to the decoding module to perform decoding through the inverse operation of encoding, and the decoded image may be input to the deep neural network-based image post-processing module (step S930). The deep neural network based image post-processing module can operate using the results learned in the deep neural network based image preprocessing module. In step S940, the deep neural network based image post-processing module may restore the blurred image using the input decoded image and the restoration parameter and / or blur level for each unit partition area (block or patch). In step S950, the restored blurred image may be rendered through the output device.

Although the exemplary methods of this disclosure are represented by a series of acts for clarity of explanation, they are not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement the method according to the present disclosure, the illustrative steps may additionally include other steps, include the remaining steps except for some steps, or may include additional steps other than some steps.

The various embodiments of the disclosure are not intended to be all-inclusive and are intended to illustrate representative aspects of the disclosure, and the features described in the various embodiments may be applied independently or in a combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays A general processor, a controller, a microcontroller, a microprocessor, and the like.

The scope of the present disclosure is to be accorded the broadest interpretation as understanding of the principles of the invention, as well as software or machine-executable instructions (e.g., operating system, applications, firmware, Instructions, and the like are stored and are non-transitory computer-readable medium executable on the device or computer.

Claims (1)

Dividing an image to be encoded into one or more unit areas;
Deriving a restoration parameter or blur level for each of the unit areas;
Performing blurring on each of the unit areas to obtain blur images; And
And encoding the blurred image,
Wherein deriving a reconstruction parameter or a blur level for each of the unit areas comprises:
A reconstruction parameter for each of the one or more sample images and the blurred image of each of the blurred images of the sample image,
Wherein the blur level for the unit area is a blur level,
Wherein the determination is made based on the complexity of the unit area.

Images