KR102103727B1 - Apparatus and method for generating image using skim-pixel convolution neural network - Google Patents

Abstract

The present application relates to a method for automatically generating an image using a scheme-pixel CNN and an apparatus for automatically generating an image. In the method for automatically generating an image using a scheme-pixel CNN according to an embodiment of the present invention, a pixel prediction unit is already generated in an image. A prediction step of simultaneously generating pixel prediction values of a plurality of target pixels to be generated using the pixel values of the existing pixels; A reliability generation step of generating a confidence for the pixel predicted values for each target pixel by a reliability estimation unit; If the reliability of the target pixel is greater than or equal to a set value, a skimming step in which the pixel generator sets the pixel prediction value to the pixel value of the target pixel; And if the reliability of the target pixel is less than the set value, the pixel generator generates a pixel inference value of the target pixel using a pixel CNN model (Pixel Convolution Neural Network), and the pixel inference value is a pixel of the target pixel. It may include a draw step of setting the value.

Description

Scheme—Automatic image generation method using pixel CN and automatic image generation device {Apparatus and method for generating image using skim-pixel convolution neural network}

The present application relates to an automatic image generation method and an automatic image generation device capable of automatically generating a new image using a learned image, and more particularly to an automatic image generation method and an automatic image generation device using a pixel CNN model.

Machine learning is a core technology used in almost all fields, such as Internet information retrieval, text mining, speech recognition, robotics, and service industry. Deep learning, a field of machine learning, has recently been spotlighted in various fields, and in the field of image-based object recognition, a convolutional neural network (CNN) is used as a kind of deep learning technology. Machine learning techniques are drawing attention.

Convolutional neural network technology, also called convolutional neural network technology, is actively researched in various image processing or computer vision fields, such as understanding the input image through calculation, extracting features, and generating new images. It is a kind of artificial neural network technology designed by simulating the nervous system of.

The present application is to provide a method for automatically generating an image and a method for automatically generating an image using a scheme-pixel CNN that can automatically generate a new image using a learned image.

This application is intended to provide a method for automatically generating an image and an apparatus for automatically generating an image using scheme-pixel CNN, which can dramatically reduce the amount of computation and time required for image generation.

In the method of automatically generating an image using the scheme-pixel CNN according to an embodiment of the present invention, the pixel prediction unit simultaneously uses the pixel values of existing pixels already generated in the image, to simultaneously generate pixel prediction values of a plurality of target pixels to be generated. A generating prediction step; A reliability generation step of generating a confidence for the pixel predicted values for each target pixel by a reliability estimation unit; If the reliability of the target pixel is greater than or equal to a set value, a skimming step in which the pixel generator sets the pixel prediction value to the pixel value of the target pixel; And if the reliability of the target pixel is less than the set value, the pixel generator generates a pixel inference value of the target pixel using a pixel CNN model (Pixel Convolution Neural Network), and the pixel inference value is a pixel of the target pixel. It may include a draw step of setting the value.

The apparatus for automatically generating an image using the scheme-pixel CNN according to an embodiment of the present invention uses the pixel values of the existing pixels already generated in the image to generate pixel prediction values of a plurality of target pixels simultaneously. part; A reliability estimation unit generating confidence for the pixel prediction values for each target pixel; And if the reliability of the target pixel is greater than or equal to a set value, the pixel predicted value is set as a pixel value of the target pixel, and if the reliability of the target pixel is less than the set value, the target is used using a pixel CNN model (Pixel Convolution Neural Network). And a pixel generating unit generating a pixel inference value of a pixel and setting the pixel inference value to a pixel value of the target pixel.

An apparatus for automatically generating an image using scheme-pixel CNN according to another embodiment of the present invention includes a processor; And a memory coupled to the processor, wherein the memory includes one or more modules configured to be executed by the processor, and the one or more modules use pixel values of existing pixels already generated in an image. , Simultaneously generating pixel prediction values of a plurality of target pixels to be generated, generating confidence for the pixel prediction values for each target pixel, and if the reliability of the target pixel is greater than or equal to a set value, the pixel prediction value is Set the pixel value of the target pixel, and if the reliability of the target pixel is less than the set value, generate a pixel inference value of the target pixel using a pixel CNN model (Pixel Convolution Neural Network), and the pixel inference value It may include a command to set the pixel value of the target pixel.

In addition, not all the features of the present invention are listed in the solution means of the above-mentioned subject. Various features of the present invention and the advantages and effects thereof may be understood in more detail with reference to specific embodiments below.

According to the method for automatically generating an image using the scheme-pixel CNN and the apparatus for automatically generating an image according to an embodiment of the present invention, a pixel value may be set as a simple prediction model for a relatively insignificant pixel region when generating an image. That is, since it is possible to omit the generation of pixel inference values using the pixel CNN model, it is possible to significantly reduce the amount of computation required. In addition, since the pixel inference value is directly generated using the pixel CNN model in a region of relatively high importance, it is possible to generate a high-quality image while reducing the amount of computation required to generate the image and improving the computation speed.

However, the effects that can be achieved by the method for automatically generating an image and the method for automatically generating an image using the scheme-pixel CNN according to embodiments of the present invention are not limited to those mentioned above, and other effects not mentioned are as follows. From the description of the present invention will be clearly understood by those of ordinary skill in the art.

1 is a schematic diagram showing an image generating apparatus according to an embodiment of the present invention.
2 and 3 are block diagrams showing an image generating apparatus according to an embodiment of the present invention.
4 to 6 are schematic diagrams showing image generation by an image generation apparatus according to an embodiment of the present invention.
7 is a view showing images generated using an image generating apparatus according to an embodiment of the present invention.
8 is a graph showing an image generation speed of an image generating apparatus according to an embodiment of the present invention.
9 is a flowchart illustrating an image generation method according to an embodiment of the present invention.

Hereinafter, exemplary embodiments disclosed herein will be described in detail with reference to the accompanying drawings, but the same or similar elements are assigned the same reference numbers regardless of the reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "modules" and "parts" for the components used in the following description are given or mixed only considering the ease of writing the specification, and do not have meanings or roles distinguished from each other in themselves. That is, the term 'unit' used in the present invention refers to a hardware component such as software, FPGA or ASIC, and 'unit' performs certain roles. However, 'wealth' is not limited to software or hardware. The 'unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, Includes subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays and variables. The functionality provided within components and 'parts' may be combined into a smaller number of components and 'parts' or further separated into additional components and 'parts'.

In addition, in describing the embodiments disclosed in the present specification, when it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed herein, detailed descriptions thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed in the specification is not limited by the accompanying drawings, and all modifications included in the spirit and technical scope of the present invention , It should be understood to include equivalents or substitutes.

1 is a schematic diagram showing an apparatus for automatically generating an image according to an embodiment of the present invention.

The image automatic generation device 100 may learn the learning images t_i using a machine learning technique such as deep learing, and random images (g_i) based on the learned results. Can be newly created. For example, after learning portrait pictures on the image automatic generation device 100 and instructing to generate a new portrait picture, the image automatic generation device 100 may generate a new portrait picture different from images already learned. have. Here, the probability distribution of pixels included in the same kind of images may be formed similarly to each other, and the image automatic generation device 100 learns a plurality of training images t_i, thereby making the images of the same kind. Probability distribution of pixels included in can be obtained.

As shown in FIG. 1, the image automatic generation device 100 may generate an image g_i including a plurality of pixels arranged in n rows and m columns. At this time, each row may be sequentially generated from top to bottom, and pixel values may be set while proceeding from left to right with respect to the pixels included in each row. Here, the image (g_i) generated by the image automatic generation device 100 may be represented as a sequence using pixel values corresponding to each pixel, for example, the image X is X = {x _i | i = 1, ..., n × m}.

Here, the image auto-generation device 100 may generate an image by using a Pixel Convolution Neural Network (CNN) model. The pixel CNN model can be expressed by the following equation.

Here, X _{≤ i} = {x ₁ , ..., x _i } is the pixel values of existing pixels already generated in the image being generated, and X _{j: i} = {x _{i + 1} , ..., x _j } The target pixels to be generated are pixel values, and p (X) corresponds to a probability function for the pixel value of the image X including n × m pixels. Further, n × m is the total number of pixels included in the image, and satisfies j> i, ∀j, i∈ [1, n × m].

At this time, in the pixel CNN model, p _θ (x _l | x ₁ , ..., x _l-1 ) can be approximated by filtering using masked convolution. Through this, the pixel CNN model can _infer the pixel value of the next pixel (X _{i + 1} ) from the pixel values of the existing pixels (X _{≤ i} ) in the image, and generate an image using the inferred pixel inference values. You can.

However, when using the pixel CNN model, it is possible to infer the pixel value of the next pixel by knowing the pixel value up to the previous pixel. Accordingly, all pixel inference values for each pixel included in the image to be generated must be calculated, and each pixel inference value must be sequentially calculated one by one. That is, although it is possible to generate a new image using the pixel CNN model, there is a large amount of computation to be performed and it may take a relatively long time to generate the image.

In order to solve this, the image automatic generation apparatus 100 according to an embodiment of the present invention uses a skim-pixel convolution neural network (CNN), for pixel regions that are relatively insignificant when generating an image. A pixel value can be set with a simple prediction model. That is, since it is possible to omit the generation of pixel inference values using the pixel CNN model, it is possible to significantly reduce the amount of computation required. In addition, since the pixel inference value is directly generated using the pixel CNN model in a region of relatively high importance, it is possible to generate a high-quality image while reducing the amount of computation required to generate the image and improving the computation speed.

2 is a block diagram showing an image automatic generation device 100 according to an embodiment of the present invention.

Referring to FIG. 2, the apparatus 100 for automatically generating an image according to an embodiment of the present invention may include a pixel prediction unit 110, a reliability estimation unit 120, and a pixel generation unit 130.

Hereinafter, an automatic image generation apparatus according to an embodiment of the present invention will be described with reference to FIG. 2.

The pixel prediction unit 110 may simultaneously generate pixel prediction values of a plurality of target pixels to be generated by using pixel values of existing pixels already generated in an image. Here, the pixel prediction unit 110 may set the target region p1 in advance, as illustrated in FIG. 4 (a), and may simultaneously generate pixel prediction values of target pixels included in the target region p1. . Then, when the setting of the pixel value for one row is completed, the next predetermined number of rows (for example, two) is set as the target region, and the pixels for each target pixel included in the target region are set. Forecasts can be generated.

Specifically, when a pixel prediction value is generated for the j-th target pixel while the pixel value is set up to the i-th pixel (j> i, ∀j, i∈ [1, n × m]), the pixel prediction unit 110 By applying the pixel values up to the i-th pixel and the pre-prediction values from the i + 1th target pixel to the j-1th target pixel in the pixel CNN model, a pixel prediction value for the j-th target pixel can be generated. have.

When using the existing pixel CNN model, in order to calculate the pixel inference value for the j-th target pixel, it was necessary to calculate all the pixel inference values up to the j-1th target pixel. On the other hand, the pixel prediction unit 110 applies the pre-prediction value from the i + 1th pixel to the j-1th pixel, so even if the pixel value up to the j-1th pixel is unknown, the It is possible to generate pixel prediction values in advance. That is, since the pixel predictor 110 can apply the pixel values of the existing pixels and the pre-prediction values in parallel to the pixel CNN model, it is possible to simultaneously calculate each pixel prediction value for a plurality of target pixels.

Here, the predicted value can be extracted using a U-net neural network. The U-net neural network has an autoencoder structure and provides an approximation to the pixel value of the i + 1th target pixel to the j-1th target pixel using the pixel values from the first pixel to the i-th pixel. can do. At this time, the pre-predicted value has the characteristics of independent identically distributed (IID), and it is possible to simultaneously calculate the pixel predicted value for the j-th target pixel and the pixel predicted value for the j + 1st target pixel. That is, since the pre-prediction values for each of the target pixels can be generated in advance, the pixel predictor 110 can apply the pixel values of the existing pixels and the pre-prediction values to the pixel CNN model in parallel. It is possible to calculate the respective pixel prediction values for the target pixels of.

Specifically, the pixel prediction unit 110 may apply Equation (1) to Equation (1) to represent Equation (2) below.

Here, q (x) is an approximate probability function of the image X to which the pre-prediction value is applied, and each pre-prediction value for the target pixels is Z _{j-1: i} = {z _{i + 1} , ..., z _{j- 1}} Z _j-1: may be defined as _{i = f w (X ≤i)} , advance predictive value may have the characteristics of the IID (independent and identically distributed) in the X _≤i. Here, f _w (X _{≤ i} ) may correspond to a U-net neural network, and p _θ (x _l | X _{≤ i} , z _{i + 1} , ..., z _l-1 ) is the same as the pixel CNN model. It can be calculated by filtering using mask convolution. Here, q (x) may be generated by learning a plurality of training images together with the pixel CNN model.

The reliability estimator 120 may generate confidence in pixel prediction values generated by the pixel prediction unit 110. Since the pixel prediction value generated by the pixel prediction unit 110 calculates pre-prediction values from the i + 1th target pixel to the j-1th target pixel in an AR (auto regressive) method, the error included in the pre-prediction value gradually increases. You can. That is, when the image is generated with only the values predicted by the pixel prediction unit 110, there is a risk of generating an image different from a desired result due to an error. Therefore, the reliability estimator 120 may calculate the reliability of whether the pixel prediction value generated by the pixel prediction unit 110 can be used and provide it as a quantitative value.

Specifically, the reliability can be calculated using Equation 3 below.

는

을 만족하며, k=i+1, ... ,j이다. 즉, 신뢰도(f_k)는 픽셀예측값(x_k)이 픽셀CNN모델을 이용하여 계산한 픽셀추론값(

)과 동일할 확률에 대응한다. 여기서, 픽셀예측값이 픽셀CNN모델을 이용하여 계산한 픽셀추론값과 동일할 확률이 높을수록 픽셀예측값에 대한 신뢰도가 높고, 확률이 낮을수록 픽셀예측값에 대한 신뢰도는 낮게 설정될 수 있다. Here, f _k is the reliability of the pixel predicted value x _k of the k th target pixel, and the pixel inference value

The

And k = i + 1, ..., j. That is, the reliability (f _k ) is the pixel inference value (x _k ) calculated by using the pixel CNN model.

). Here, the higher the probability that the pixel prediction value is the same as the pixel inference value calculated using the pixel CNN model, the higher the reliability of the pixel prediction value, and the lower the probability, the lower the reliability of the pixel prediction value.

The reliability estimator 120 can learn the difference between the pixel inference value generated by the pixel CNN model and the pixel prediction value generated by the pixel prediction unit 110 by machine learning techniques such as deep learning, and reliability according to the learned model. Can be calculated. According to an embodiment, after generating a sample image using the pixel CNN model, learning a difference between a pixel inference value of each pixel included in the generated sample image and a pixel prediction value generated by the pixel prediction unit 110 can do.

Additionally, depending on the embodiment, it is also possible to display the reliability by binary classification. That is, when the reliability is higher than the set value (attention threshold), the pixel predicted value can be determined as a reliable case to reset the reliability to 1, and when the reliability is less than the set value, the pixel predicted value is determined as unreliable, and the reliability is determined. Can be reset to zero.

The pixel generator 130 may set the pixel value of the target pixel according to the reliability of the pixel predicted value for the target pixel. That is, if the reliability of the target pixel is greater than or equal to the set value, the pixel predicted value is set as the pixel value of the target pixel, and if the reliability of the target pixel is less than the set value, the pixel inference value generated by the pixel CNN model can be set as the pixel value of the target pixel. have.

As illustrated in FIG. 4, when the target region P1 is set, the pixel prediction unit 110 and the reliability estimator 120 may generate pixel prediction values and reliability corresponding to the target region P1. Here, the reliability may be displayed as a binarization image based on a set value. That is, when the reliability is higher than the set value, it may be displayed as white (1), and when the reliability is lower than the set value, it may be represented as black (0).

Thereafter, as illustrated in FIG. 5 (a), pixel values may be sequentially set from pixel values of target pixels included in a1 area. Here, in FIG. 5 (a), since the reliability corresponding to the area a1 is displayed in white, this corresponds to a case where the pixel predicted value is reliable. Therefore, the pixel value for the area a1 can be set according to the pixel prediction value.

In addition, the pixel values of the target pixels included in the area a2 corresponding to the area next to the area a1 can be set as shown in FIG. 5 (b). That is, since the reliability corresponding to the region a2 is displayed in black, it corresponds to a case where the pixel prediction value is not reliable. Therefore, pixel prediction values corresponding to the area a2 may not be applied to the area a2. Instead, pixel inference values may be calculated using the pixel CNN model, and the calculated pixel inference values may be set as pixel values in the a2 region. In this case, as shown in Fig. 6 (a), pixel values in the area a2 can be set.

Meanwhile, as shown in FIG. 6 (a), when the pixel value of the target pixel is set as the pixel inference value, the pixel prediction value and the reliability of the remaining target area p2 may be calculated and updated again. That is, by updating the pixel prediction value and reliability by reflecting the pixel value of the target pixel set as the pixel inference value, errors included in the existing pixel prediction value can be removed and more accurate pixel prediction value and reliability can be generated.

Additionally, the pixel generator 130 may generate first k pixels for image generation in advance. That is, for the first k pixels, the pixel inference value may be calculated using only the pixel CNN model, and an image may be generated by using the pixel CNN model. In some cases, it is also possible to assign random pixel values to the first k pixels. For example, up to the first three columns when creating an image, pixel values can be set as pixel inference values or random values using the pixel CNN model.

The pixel generator 130 may repeatedly perform the above-described processes until image generation is completed.

Meanwhile, as illustrated in FIG. 3, the apparatus 100 for automatically generating an image according to an embodiment of the present invention may include physical configurations such as a processor 10 and a memory 40, and the memory 40 ), One or more modules configured to be executed by the processor 10 may be included. Specifically, the one or more modules may include a pixel prediction module, a reliability estimation module, and a pixel generation module.

The processor 10 may perform various functions and execute data processing functions by executing various software programs and a set of instructions stored in the memory 40. The peripheral interface unit 30 may connect the input / output peripheral devices of the image automatic generation device 100 to the processor 10 and the memory 40, and the memory controller 20 may include the processor 10 or the image automatic generation device ( When a component of 100) accesses the memory 40, a function of controlling memory access may be performed. Depending on the embodiment, the processor 10, the memory controller 20 and the peripheral interface unit 30 may be implemented on a single chip or may be implemented as separate chips.

The memory 40 may include a high-speed random access memory, one or more magnetic disk storage devices, and non-volatile memory such as a flash memory device. In addition, the memory 40 may further include a storage device located away from the processor 10 or a network attached storage device accessed through a communication network such as the Internet.

The display unit 50 may be configured to display the user so that the image generated through time can be checked. For example, the display unit 50 includes a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, and a flexible display. , 3D display (3D display), can be displayed visually using an electrophoretic display (electrophoretic display). However, the content of the present invention is not limited to this, and the display unit may be implemented in various ways.

The input unit 60 receives input from a user, and includes a keyboard, a keypad, a mouse, a touch pen, a touch pad, a touch panel, and a jog A jog wheel, a jog switch, and the like may correspond to the input unit 60.

On the other hand, as shown in Figure 3, the automatic image generating apparatus 100 according to an embodiment of the present invention, the memory 40, the operating system, the pixel prediction module, reliability estimation module and pixel corresponding to the application program It may include a generation module. Here, each module is a set of instructions for performing the above-described functions, and may be stored in the memory 40.

Therefore, in the terminal device 100 according to an embodiment of the present invention, the processor 10 may access the memory 40 and execute instructions corresponding to each module. However, the pixel prediction module, the reliability estimation module, and the pixel generation module correspond to the pixel prediction unit, the reliability estimation unit, and the pixel generation unit, respectively, so detailed descriptions thereof are omitted here.

7 is an example showing images generated using a scheme-pixel CNN according to an embodiment of the present invention. In FIG. 7, the left column indicates the area to which the pixel predicted value is applied (white), and the center column indicates the reliability of the image, and the higher the reliability, the red and the lower the reliability, the blue. The last right column corresponds to the actual created image. In addition, FIG. 7 (a) shows that the entire image is generated using pixel prediction values, and FIG. 7 (f) shows that the entire image is generated as pixel inference values using the pixel CNN model. The setting value for applying the pixel CNN model was set higher as it went to 7 (f). Referring to FIG. 7, in the case of a portrait image, it can be confirmed that the reliability of the characteristic part of the character, such as the aspect ratio, is set relatively low, and the difference in image quality is compared to the case where only the pixel CNN model is used. It can be confirmed that it does not appear significantly. In addition, as shown in FIG. 8, it can be seen that the higher the ratio using the pixel prediction value, the faster the image creation speed.

9 is a flowchart illustrating an automatic image generation method according to an embodiment of the present invention.

Referring to FIG. 9, the automatic image generation method according to an embodiment of the present invention includes an initial generation step (S10), a prediction step (S20), a reliability generation step (S30), a skimming step (S40), and a draw step (S50). And it may include an update step (S60).

Hereinafter, a method for automatically generating an image according to an embodiment of the present invention will be described with reference to FIG. 9.

In the initial generation step (S10), the pixel generating unit may generate the first k pixels for the image to be generated using the pixel inference value extracted from the pixel CNN model. That is, for the first k pixels, the pixel inference value may be calculated using only the pixel CNN model, and an image may be generated by using the pixel CNN model. In some cases, it is also possible to assign random pixel values to the first k pixels. For example, up to the first three columns when creating an image, pixel values can be set as pixel inference values or random values using the pixel CNN model.

In the prediction step S20, the pixel prediction unit may simultaneously generate pixel prediction values of a plurality of target pixels to be generated by using pixel values of existing pixels already generated in an image. Here, the pixel prediction unit may set a target region in advance, and simultaneously generate pixel prediction values of target pixels included in the target region. Then, when the setting of the pixel value for one row is completed, the next predetermined number of rows (for example, two) is set as the target region, and the pixels for each target pixel included in the target region are set. Forecasts can be generated.

Specifically, when a pixel prediction value is generated for the j-th target pixel while the pixel value is set up to the i-th pixel (j> i, ∀j, i∈ [1, n × m]), the pixel prediction unit is the i-th A pixel prediction value for the j-th target pixel may be generated by applying the pixel values up to the pixel and the pre-prediction values from the i + 1st target pixel to the j-1th target pixel in the pixel CNN model. Here, the predicted value may be extracted using a U-net neural network, and the predicted value may have the characteristics of independent identically distributed (IID). That is, since the pre-prediction values for each of the target pixels can be generated in advance, the pixel predictor can apply the pixel values of the existing pixels and the pre-prediction values to the pixel CNN model in parallel, through which a plurality of target pixels It is possible to calculate the respective pixel prediction values for.

In the reliability generation step (S30), the reliability estimator may generate confidence for the pixel prediction values for each target pixel. That is, the reliability estimator may calculate the reliability of whether the pixel prediction value generated by the pixel prediction unit can be used and provide it as a quantitative value. Here, the reliability may be a probability that a pixel prediction value for a target pixel coincides with a pixel inference value for the target pixel, and the reliability estimator learns the difference between the pixel inference value and the pixel prediction value using a machine learning technique such as deep learning. Can be calculated. Specifically, after generating a sample image using the pixel CNN model, it can be learned by comparing the pixel inference value of each pixel included in the generated sample image with the pixel prediction value generated by the pixel prediction unit.

In the skimming step (S40), the reliability of the target pixel may be compared with the set value, and if the reliability is greater than or equal to the set value, the pixel generator may set the pixel prediction value as the pixel value of the target pixel. That is, reliability can be sequentially determined for each target pixel included in the target area, and a pixel value can be set as a pixel prediction value for a target pixel having a reliability greater than or equal to a set value. In this case, since the pixel inference value is not calculated using the pixel CNN model, the pixel value can be quickly set. On the other hand, if the reliability of the target pixel is less than the set value, the process may proceed to the drawing step S50.

In the drawing step (S50), if the reliability of the target pixel is less than the set value, the pixel generator generates a pixel inference value of the target pixel using a pixel CNN model (Pixel Convolution Neural Network), and the pixel inference value is a pixel of the target pixel. Can be set as a value. That is, since the reliability of the pixel prediction value of the target pixel is low, the pixel value can be set as the pixel inference value using the pixel CNN model instead of the pixel prediction value. Here, when the pixel inference value of the target pixel is calculated using the pixel CNN model, the calculation time may be relatively long, but more accurate image generation is possible.

Meanwhile, after the pixel value of the target pixel is set as the pixel inference value, an update step S60 of calculating and updating the pixel predicted value and reliability of the remaining target area again may be performed. That is, by updating the pixel prediction value and reliability by reflecting the pixel value of the target pixel set as the pixel inference value, errors included in the existing pixel prediction value can be removed and more accurate pixel prediction value and reliability can be generated.

Thereafter, the skimming step (S40), the draw step (S50), and the update step (S60) may be repeated until image generation is completed to generate an image.

The above-described present invention can be embodied as computer readable codes on a medium on which a program is recorded. The computer-readable medium may be a computer that continuously stores executable programs or may be temporarily stored for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combinations, and is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks, And program instructions including ROM, RAM, flash memory, and the like. In addition, examples of other media include an application store for distributing applications, a site for distributing or distributing various software, and a recording medium or storage medium managed by a server. Accordingly, the above detailed description should not be construed as limiting in all respects, but should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

The present invention is not limited by the above-described embodiments and the accompanying drawings. For those skilled in the art to which the present invention pertains, it will be apparent that components according to the present invention can be substituted, modified and changed without departing from the technical spirit of the present invention.

100: automatic image generation device 110: pixel prediction unit
120: reliability estimation 130: pixel generation department
S10: Initial generation stage S20: Prediction stage
S30: Reliability creation step S40: Skimming step
S50: Draw stage S60: Update stage

Claims (14)

In the method of automatically generating an image using a Skim-PixelCNN (Skim-Pixel Convolution Neural Network),
A prediction step of generating a pixel prediction value of a plurality of target pixels to be generated by using the pixel prediction unit using the pixel values of the existing pixels already generated in the image and the pixel prediction values in the image in which the pixel values are not determined. ;
A reliability generation step of generating a confidence for the pixel predicted values for each target pixel by a reliability estimation unit;
If the reliability of the target pixel is greater than or equal to a set value, a skimming step in which the pixel generator sets the pixel prediction value to the pixel value of the target pixel; And
If the reliability of the target pixel is less than the set value, the pixel generator generates pixel inference values of the target pixel by applying the pixel values up to the pixel immediately before the target pixel to the pixel CNN model (Pixel Convolution Neural Network), And a drawing step of setting the pixel inference value to a pixel value of the target pixel.
The method of claim 1, wherein the image
It is created to include a plurality of pixels arranged in n rows and m columns.
Each row is sequentially generated from top to bottom, and the pixel values of the pixels included in the row are generated while proceeding from left to right.
The method of claim 1, wherein the prediction step
When the setting of a pixel value for one row is completed, an automatic image generation method characterized by generating pixel prediction values for a target region including a predetermined number of rows at the same time.
According to claim 3,
When the pixel value of the target pixel is set to the pixel inference value, further comprising an update step of reflecting the pixel value of the target pixel and updating the pixel prediction value and reliability of the remaining target area after the target pixel. A method for automatically generating an image.
In the method of automatically generating an image using a Skim-PixelCNN (Skim-Pixel Convolution Neural Network),
A prediction step in which the pixel prediction unit simultaneously generates pixel prediction values of a plurality of target pixels to be generated by using pixel values of existing pixels already generated in an image;
A reliability generation step of generating a confidence for the pixel predicted values for each target pixel by a reliability estimation unit;
If the reliability of the target pixel is greater than or equal to a set value, a skimming step in which the pixel generator sets the pixel prediction value to the pixel value of the target pixel; And
If the reliability of the target pixel is less than the set value, the pixel generator generates a pixel inference value of the target pixel using a pixel CNN model (Pixel Convolution Neural Network), and the pixel inference value is a pixel value of the target pixel Including the draw (draw) step to set,
The image above
It is created to include a plurality of pixels arranged in n rows and m columns.
Each row is sequentially generated from top to bottom, and pixel values of pixels included in the row are generated while proceeding from left to right,
The prediction step
When the pixel prediction value for the j-th target pixel is generated while the pixel value is set up to the i-th pixel (j> i, ∀j, i∈ [1, n × m]), the pixel values up to the i-th pixel And generating a pixel prediction value for the j-th target pixel by applying a pre-prediction value from the i + 1st target pixel to a j-1th target pixel in the pixel CNN model.
The method of claim 5, wherein the prediction step
A method for automatically generating an image, wherein the predicted values from the i + 1 th target pixel to the j-1 th target pixel are respectively extracted using a U-net neural network.
The method of claim 6, wherein the prediction step
A method of automatically generating an image, wherein the pixel values of the existing pixels and the pre-prediction values are applied in parallel to the pixel CNN model, and the respective pixel prediction values for a plurality of target pixels are calculated at the same time.
In the method of automatically generating an image using a Skim-PixelCNN (Skim-Pixel Convolution Neural Network),
A prediction step in which the pixel prediction unit simultaneously generates pixel prediction values of a plurality of target pixels to be generated by using pixel values of existing pixels already generated in an image;
A reliability generation step of generating a confidence for the pixel predicted values for each target pixel by a reliability estimation unit;
If the reliability of the target pixel is greater than or equal to a set value, a skimming step in which the pixel generator sets the pixel prediction value to the pixel value of the target pixel; And
If the reliability of the target pixel is less than the set value, the pixel generator generates a pixel inference value of the target pixel using a pixel CNN model (Pixel Convolution Neural Network), and the pixel inference value is a pixel value of the target pixel Including the draw (draw) step to set,
The reliability estimation unit
Using the sample image generated using the pixel CNN model, learning and generating a difference between a pixel inference value of each pixel included in the sample image and a pixel prediction value generated by the pixel prediction unit. How to automatically generate images.
The method of claim 8, wherein the reliability estimation unit
A method for automatically generating an image, wherein a probability that a pixel prediction value for the target pixel matches a pixel inference value for the target pixel is calculated and provided as a reliability for the target pixel.
The method of claim 8, wherein the pixel prediction unit
A method for automatically generating an image, characterized in that learning is performed using a plurality of training images together with the pixel CNN model.
According to claim 1,
And the pixel generating unit further comprises an initial generation step of generating the first k pixels of the image by using the pixel inference value extracted from the pixel CNN model.
A computer program stored in a medium in combination with hardware to execute the automatic image generation method of any one of claims 1 to 11.
A pixel prediction unit generating a pixel prediction value of a plurality of target pixels to be generated by using a pixel value of existing pixels already generated in an image and a pre-prediction value for each pixel whose pixel value is not determined in the image;
A reliability estimation unit generating confidence for the pixel prediction values for each target pixel; And
If the reliability of the target pixel is greater than or equal to the set value, the pixel predicted value is set as the pixel value of the target pixel, and if the reliability of the target pixel is less than the set value, pixel values up to the immediately preceding pixel of the target pixel are pixel CNN models ( Skim-PixelCNN (Skim-Pixel Convolution) including a pixel generator that applies to a Pixel Convolution Neural Network to generate a pixel inference value of the target pixel and sets the pixel inference value to a pixel value of the target pixel. Neural Network).
Processor; And
Including a memory coupled to the processor,
The memory includes one or more modules configured to be executed by the processor,
The one or more modules,
A pixel prediction value of a plurality of target pixels to be generated is generated by using a pixel value of existing pixels already generated in the image and a pre-prediction value for each pixel in which the pixel value is not determined in the image,
Confidence of the pixel prediction values is generated for each target pixel,
If the reliability of the target pixel is greater than or equal to a set value, the pixel predicted value is set to the pixel value of the target pixel,
If the reliability of the target pixel is less than the set value, the pixel inference value of the target pixel is generated by applying the pixel values up to the immediately preceding pixel of the target pixel to the pixel CNN model (Pixel Convolution Neural Network), and the pixel inference value Setting the pixel value of the target pixel,
An apparatus for automatically generating an image using a Skim-PixelCNN (Skim-Pixel Convolution Neural Network) including a command.

Images