KR102280201B1 - Method and apparatus for inferring invisible image using machine learning - Google Patents

Abstract

It relates to a method and apparatus for inferring a hidden image using machine learning, wherein at least two data of at least one image, at least one text, and at least one feature value is input, and a generative network is constructed through learning. Generates at least one image inferred from at least two data input according to the correlation model between images, the classification network determines the suitability score of each inferred image, and compares the suitability score of each inferred image with the reference score Outputs each inferred image as a hidden image between the images represented by at least two input data based on .

Description

{Method and apparatus for inferring invisible image using machine learning}

It relates to a method and apparatus for automatically generating an image using machine learning.

Machine learning, which implements artificial intelligence in software, is an algorithm in which a computer learns data, finds patterns on its own, and learns to perform appropriate tasks. Machine learning is largely classified into supervised learning, unsupervised learning, reinforcement learning, and the like.

Supervised learning refers to an algorithm that learns from a given correct answer. The purpose of supervised learning is classification and regression. Classification means classifying certain data into a finite number of classes. Specifically, it is divided into binary classification, which classifies data into two classes, and multiclass classification, which classifies data into three or more classes. Regression generation refers to predicting successive values based on the characteristics of data and is used to predict successive numbers. Regression generation is used to create completely new images or synthesize voices (Text-to-Speech) using deep learning.

Unsupervised learning is an algorithm that learns without an answer given, and a representative unsupervised learning includes clustering. Attempts to predict situations based on data with such supervised and unsupervised learning continue, but supervised learning has a limit on the amount of data that can be used because only data with a given correct answer can be used. Supervised learning is becoming more and more important, and a representative example of such unsupervised learning is Generative Adversarial Networks (GAN).

Korean Patent Application Laid-Open No. 10-2017-0137350 "Apparatus and method for learning object movement pattern using neural network generation model" acquires movement patterns classified by topic by tracking feature points of moving objects in an input image and learning each movement pattern Disclosed is an apparatus for learning an object movement pattern using a neural network generation model including a unit and a movement pattern learning unit for each image that learns a movement pattern according to an input image using movement pattern information stored in the movement pattern acquisition unit.

However, this prior art receives a CCTV (Closed Circuit Television) camera image of about a day for a monitoring area as a learning image, and after learning using DBN (Deep Belief Networks) and GAN (Generative Adversarial Networks), the monitoring area It is possible to check what movement pattern is appearing by receiving the video of the image, but it cannot provide an image of an invisible object in a blind spot that cannot be captured by CCTV or is obscured by obstacles such as a building.

An object of the present invention is to provide a method and apparatus for inferring a hidden image that can provide an image of an invisible object in a blind spot that cannot be seen by an obstacle such as a building or can not be photographed by CCTV. Another object of the present invention is to provide a computer-readable recording medium in which a program for executing the method in a computer is recorded. It is not limited to the technical problems as described above, and another technical problem may be derived from the following description.

A hidden image inference method according to an aspect of the present invention comprises: inputting at least two data of at least one image, at least one text, and at least one feature value; generating, by the network, generating at least one image inferred from the input at least two data according to a correlation model between images built through learning; determining, by the classification network, a suitability score of each of the generated inference images; and outputting each of the generated inference images as a hidden image between images represented by the input at least two data based on a comparison result between the suitability score and the reference score of the generated inference image.

The determining of the suitability score may include determining the suitability score of each of the inferred images from the input at least two pieces of data.

The determining of the suitability score includes at least two data of at least one image, at least one text, and at least one feature value generated by inputting each of the generated inference images to the generation network and the input at least two The suitability score may be determined according to the similarity of the data of dogs.

The generating of the inferred image may include: transforming at least two data of the at least one image, at least one text, and at least one feature value into latent vectors; calculating a latent vector located between the transformed latent vectors by interpolating the transformed latent vectors; and generating at least one image inferred from the at least two input data using the transformed latent vectors and the calculated latent vector.

The generating of the inferred image may include: converting each input image into a latent vector; and inferring characteristics of a correlation between images represented by the at least two input data from the viewpoint of each input image by using the transformed latent vector.

The generating of the inference image may include: converting each inputted text into a latent vector; and inferring characteristics of a correlation between images represented by the at least two input data from the viewpoint of each input text using the transformed latent vector.

The generating of the inferred image may include: converting each of the input feature values into a latent vector; and inferring characteristics of a correlation between images represented by the at least two input data from the viewpoint of each input feature value by using the transformed latent vector.

The generating of the inferred image may include: converting the input at least one image, at least one text, and at least one feature value into latent vectors; inferring characteristics of a correlation between images represented by the at least two input data using the transformed latent vectors; and generating at least one image inferred from the input at least two data by integrating the inferred features and generating at least one image inferred from the integrated features according to the association model. can do.

According to another aspect of the present invention, there is provided a computer-readable recording medium in which a program for executing the method for estimating the hidden image in a computer is recorded.

A hidden image reasoning apparatus according to another aspect of the present invention includes an input module for inputting at least two data of at least one image, at least one text, and at least one feature value; a generation network for generating at least one image inferred from the input at least two data according to a correlation model between images built through learning; a classification network that determines a suitability score of each of the generated inference images; and an output module for outputting each of the generated inference images as a hidden image between images represented by the input at least two data based on a comparison result of a suitability score and a reference score of the generated inferred image.

At least two data of at least one image, at least one text, and at least one feature value is input, and the generating network is inferred from the at least two data input according to the relation model between images built through learning. At least one image is generated, the classification network determines a suitability score of each inferred image, and based on the comparison result of the suitability score of each inferred image and the reference score, each inferred image is an image represented by at least two input data. By outputting it as a hidden image in between, it is possible to provide an image of an invisible object in a blind spot that cannot be photographed by an obstacle such as a building or by using the surrounding image, text, and image feature values.

In particular, even when there is only one input image, when text describing a scene of an image associated with the input image or a feature value of an image is input instead of an image, a hidden image associated with the input image may be inferred. Furthermore, even in the absence of an input image, a hidden image may be inferred from a text or image feature value describing a certain scene.

1 is a block diagram of a hidden image inference apparatus according to an embodiment of the present invention.
FIG. 2 is an exemplary diagram of the classification network 30 shown in FIG. 1 .
FIG. 3 is an exemplary diagram of the generating network 40 shown in FIG. 1 .
4 is a flowchart of a hidden image inference method according to an embodiment of the present invention.
FIG. 5 is a conceptual diagram for explaining a method of determining an image suitability score of the classification network 30 shown in FIG. 1 .
FIG. 6 is an exemplary diagram in which all data input in step 1000 shown in FIG. 4 are images.
FIG. 7 is an exemplary diagram of a case in which a part of data input in step 1000 shown in FIG. 4 is a feature value.
FIG. 8 is an exemplary diagram when a part of data input in step 1000 shown in FIG. 4 is text.
FIG. 9 is an exemplary diagram of a case in which a part of data input in step 1000 shown in FIG. 4 is a feature value and text.
FIG. 10 is a data processing process diagram of the generating network 40 for the example shown in FIG. 6 .
FIG. 11 is a data processing process diagram of the generating network 40 for the example shown in FIG. 10 .
FIG. 12 is a detailed diagram of feature value processing and text processing shown in FIG. 4 .
13 is a diagram illustrating an example of learning of the classification network 30 and the generating network 40 shown in FIG. 1 .
14 is a diagram illustrating an example of application of the hidden image inference method shown in FIG. 4 .

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a block diagram of a hidden image inference apparatus according to an embodiment of the present invention. 1, the hidden image reasoning apparatus according to the present embodiment includes an input module 10, a control module 20, a discriminative network 30, a generative network 40, a storage ( 50 ), and an output module 60 . In the method of inferring a hidden image using deep learning according to an embodiment of the present invention, when all of the original image 1 is input, the input module 10 divides the input data 1 into the classification network 30 and It outputs to the generating network 40, and according to the control of the control module 20, the classification network 30 and the generating network 40 infer and generate the hidden image 1', and the output module 60 generates it in this way. Outputs the generated hidden image.

Each of the classification network 30 and the generative network 40 shown in FIG. 1 means a neural network corresponding to a classifier of a Generative Adversarial Networks (GAN) and a neural network corresponding to a generator. The basic composition of the GAN is based on the paper "Generative Adversarial Networks" (published in 2014, Goodfellow, Ian J. ; Pouget-Abadie, jean; Mirza, Mehdi; Xu, Bing; Warde-Faeley, David; Ozair, Sherjil; Courville, Aaron; Bengio and Yoshua co-authors). In adversarial learning of GANs, the classification network is first trained, and then the process of learning the generative network is repeated while giving and receiving.

Classification network training consists of two main steps. The first step is the process of inputting real data into the classification network and training the classification network to classify the data as real. In the second step, contrary to the first step, the fake data generated by the generative network is input into the classification network. This is the process of learning to classify the data as fake. Through this process, the classification network can classify real data as real and fake data as fake. After training the classification network, the generative network is trained in the direction of deceiving the learned classification network. The fake data created by the generative network is input into the classification network, and the generative network is trained to generate data similar to real data enough to classify the fake data as real.

If this learning process is repeated, the classification network and the generative network recognize each other as hostile competitors and both develop. As a result, the generative network can create fake data perfectly similar to the real data. Fake data cannot be distinguished. That is, in the GAN, the generation network tries to lower the probability of successful classification, and the classification network tries to increase the probability of successful classification, while forming a structure in which each other competitively develops.

More specifically, the GAN learns by solving a minmax problem using the objective function V(D,G) as shown in Equation 1 below.

Here, x~p _data (x) means data sampled from a probability distribution with respect to actual data, and z~p _z (z) means data sampled from random noise using a Gaussian distribution in general. . z is usually called a latent vector, which means a vector in the latent space that can describe data well with reduced dimensions. Since D(x) is a classification network and a value between 0 and 1, which means the probability of being real, D(x) yields a value of 1 if the data is real and 0 if it is fake. G(z) is a generative network, and D(G(z)) yields a value of 1 if it is determined that G(z), the data generated by the generative network, is real, and 0 if it is judged to be fake.

Considering from the point of view that the classification network maximizes V(D,G), in order to maximize the calculated value of Equation 1, both the first term and the second term on the right side must be maximal, -D(G(z))) must all be maximal. Therefore, D(x) should be 1, which means training the classification network to classify real data as real. Similarly, 1-D(G(z) becomes 1, so D(G(z) must therefore be 0), which means training the classification network to classify the fake data generated by the generative network as fake. Learning the classification network so that (D, G) is maximized is the process of learning the classification network to classify real data as real and fake data as fake.

Next, thinking from the perspective of how the generative network learns to minimize V(D,G), since G is not included in the first term on the right side of Equation 1, it is not related to the generative network and can be omitted. To minimize the second term, log(1-D(G(z))) must be minimized. So log(1-D(G(z))) should be 0 and D(G(z)) should be 1. This is not enough to train the generative network to generate fake data that is perfect enough to classify as true. In this way, learning the classification network in the direction of maximizing V(D,G) and learning the generative network in the direction of minimizing V(D,G) is called the Minmax problem.

In order to prevent the description of the embodiment according to the present invention from being tedious, the description of known techniques in the technical field to which this embodiment belongs, such as the basic configuration of the classification network 30 and the generating network 40 described above, will be omitted and hereinafter will be omitted. The present embodiment will be described with a focus on the characteristics of the present embodiment that are different from the prior art.

FIG. 2 is an exemplary diagram of the classification network 30 illustrated in FIG. 1 , and FIG. 3 is an exemplary diagram of the generation network 40 illustrated in FIG. 1 . 2 and 3 , the classification network 30 and the generation network 40 may be implemented as a Convolutional Neural Network (CNN)-based deep learning neural network having a structure similar to each other. As described below, the generating network 40 generates a hidden image inferred from data of at least two of at least one image, at least one text, and at least one feature value, and the classification network 30 generates The suitability score of the image generated by the network 40 is output.

4 is a flowchart of a hidden image inference method according to an embodiment of the present invention. Referring to FIG. 4 , the hidden image inference method according to the present embodiment consists of the following steps that are time-series executed in the hidden image inference apparatus shown in FIG. 1 .

In step 1000 , the input module 10 generates at least two data among at least one image, at least one text, and at least one feature value according to the control of the control module 20 into the classification network 30 and the generating network 40 . ) for each. Such data is input to each of the classification network 30 and the generating network 40 through the input module 10 according to a user's operation such as data upload. Here, the image input to the generating network 40 means data of an image acquired by a user. For example, when a user can photograph an object from a partial angle using a camera, the image input to the generating network 40 may be data of an image obtained by photographing the object from the partial angle. In this case, the hidden image is an image expected to be acquired when the object is photographed from an angle that cannot be photographed.

The text input to the generating network 40 means text describing a scene represented by an image. The feature value input to the generating network 40 means a value representing the features of a certain image. In this embodiment, one text or feature value serves to replace one image. According to the present embodiment, even in a situation in which the user can obtain only one image for a certain object or the image cannot be obtained, the user inputs text or feature values instead of the image through the input module 10 to generate the network 40 ) to obtain a hidden image.

When several images have spatial and temporal correlation with each other with respect to an event, the correlation between these images is inferred from data of at least two of at least one image, at least one text, and at least one feature value. can be For example, from two images taken at two angles for a certain object, with respect to an event of taking an object in a certain space one after another over 360 degrees, a 360-degree spatial relationship centered on the object and the relationship of the object A temporal correlation of photographing changes one after another can be inferred.

When the input module 10 inputs a plurality of images to the generating network 40 , a correlation between the images may be inferred by the generating network 40 . When the input module 10 inputs a plurality of texts to the generating network 40 , a correlation between images may be inferred. When the input module 10 inputs a plurality of feature values to the generating network 40 , a correlation between images may be inferred. When the input module 10 inputs at least one image and at least one text to the generating network 40 , a correlation between the images may be inferred. When the input module 10 inputs at least one image and at least one feature value to the generating network 40 , a correlation between the images may be inferred. When the input module 10 inputs at least one text and at least one feature value to the generating network 40 , a correlation between images may be inferred.

In step 2000, when at least one image input in step 1000 exists, the generating network 40 processes each image input in step 1000 through an embedding layer of each image input in step 1000 in a convolutional layer. Convert it to a stacked image of , and convert that stack image to a latent vector representing it. Next, the generation network 40 uses the latent vector transformed as described above through a plurality of convolutional layers to determine the correlation between the images represented by the at least two data input in step 1000 from the viewpoint of each image input in step 1000. Infer features. When the suitability score for a certain inferred image is fed back from the classification network 30 to the generating network 40 , the generating network 40 is a latent vector in the direction that the suitability score of the inferred image fed back from the classification network 30 is improved. update the value of By updating the latent vector, the latent vector can better represent the features of the image in the relation between the images. Through this process, the generating network 40 is learned. As the learning amount of the generating network 40 increases, the latent vector can more and more better represent the features of the image in the relation between the images.

In step 2000, when at least one text input in step 1000 exists, the generating network 40 represents each text input in step 1000 through an embedding layer of each text input in step 1000, a latent vector representing this. convert to Next, the generation network 40 uses the latent vector converted as described above through the text processing layer to determine the characteristics of the correlation between the images represented by the at least two data input in step 1000 from the viewpoint of each text input in step 1000. infer When the suitability score for a certain inferred image is fed back from the classification network 30 to the generating network 40 , the generating network 40 is a latent vector in the direction that the suitability score of the inferred image fed back from the classification network 30 is improved. update the value of By updating the latent vector, the latent vector can better represent the characteristics of the text in the relation between images. Through this process, the generating network 40 is learned. As the learning amount of the generating network 40 increases, the latent vector can more and more better represent the characteristics of the corresponding text in the relation between images.

In step 2000, when at least one feature value input in step 1000 exists, the generating network 40 represents each feature value input in step 1000 through an embedding layer of each feature value input in step 1000, a latent vector representing this. convert to Next, the generation network 40 uses the latent vector converted as described above through the feature value processing layer to determine the correlation between the images represented by the at least two data input in step 1000 from the viewpoint of each feature value input in step 1000. Infer features. When the suitability score for a certain inferred image is fed back from the classification network 30 to the generating network 40 , the generating network 40 is a latent vector in the direction that the suitability score of the inferred image fed back from the classification network 30 is improved. update the value of By updating the latent vector, the latent vector can better represent the features of the corresponding feature value in the relation between images. Through this process, the generating network 40 is learned. As the learning amount of the generating network 40 increases, the latent vector can more and more better represent the features of the corresponding feature values in the relation between images.

According to the present embodiment, as the learning of the generating network 40 proceeds, the latent vector is the corresponding image, the corresponding text, and the corresponding image in the relation between the images due to the hostile competitive relationship between the generating network 40 and the classification network 30 . As the features of the feature value can be represented more and more better, the accuracy of the hidden image provided by the present embodiment becomes higher and higher.

As described above, the generating network 40 converts at least two pieces of data input in step 1000 into latent vectors as described above. The generation network 40 calculates at least one latent vector located between the latent vectors by interpolating the transformed latent vectors in this way, and uses the transformed latent vectors and the calculated at least one latent vector in 1000 steps. At least one image inferred from at least two pieces of data input in may be generated. Since the latent vectors transformed in step 2000 correspond to images acquired by the user, there is a high probability that at least one latent vector calculated by their interpolation corresponds to hidden images between images acquired by the user. .

That is, the latent vector calculated by interpolation of the latent vectors transformed in step 2000 serves as a larger number of input images close to the hidden image, and the accuracy of the features of the correlation between images inferred using these goes high As a result, it is possible to significantly improve the accuracy of the hidden image output from the generation network 40 through a simple operation of interpolation of the latent vectors transformed in step 2000 .

In step 3100, the generative network 40 integrates the plurality of features inferred in step 2000 through the full access layer, and the features integrated in this way according to the correlation model between the images built through the learning of the generative network 40 By generating images inferred from the data, images inferred from the at least two data input in step 1000 are generated. The generating network 40 has very poor ability to understand the relationship between several images (text and feature values are possible instead of images) at the initial learning stage. As the amount of learning increases due to the hostile competition relationship between the generative network 40 and the classification network 30, the generative network 40 and the classification network 30 can build a neural network model that can more accurately understand the relationship between them. there will be

In step 3200 , the generation network 40 generates an inference image having a level of resolution required by a user by upsampling the inferred images generated in step 3100 . In step 3300 , the generation network 40 inputs each inference image generated in step 3200 to the classification network 30 . In step 3400 , the classification network 30 determines a suitability score of each inferred image generated in step 3200 from at least two pieces of data input in step 1000 under the control of the control module 20 . Step 3400 will be described in detail below.

In step 3500, the control module 20 compares the suitability score of each inferred image determined in step 3400 with a preset reference score. As a result of the comparison, if the suitability score of each inferred image determined in step 3400 is less than the preset reference score, proceed to step 3600, and if the suitability score of each inferred image and the preset reference score are greater than or equal to the suitability score of each inference image, proceed to step 3700.

In step 3600 , the control module 20 feeds back the suitability score of each inferred image from the classification network 30 to the generating network 40 by inputting the suitability score of each inferred image to the generating network 40 . Here, the reference score may be set by the user, and as the value increases, the accuracy of each hidden image may be improved, but the number of hidden images provided according to the present embodiment may decrease. Meanwhile, as the reference score decreases, the number of hidden images provided according to the present embodiment may increase, but the accuracy of each hidden image may decrease. According to the present embodiment, the number and quality of the hidden images desired by the user may be provided according to the adjustment of the user's reference score.

In step 3700 , the control module 20 separates an inference image having a suitability score equal to or greater than the reference score from the inference image generated in step 3300 , and then stores it in the storage 50 . The output module 60 outputs each inferred image stored in the storage 50 as a hidden image between the images indicated by the at least two data input in step 1000 under the control of the control module 20 . The hidden image is provided to the user through the output module 60 such as a display panel according to the user's output command or the like.

FIG. 5 is a conceptual diagram for explaining a method of determining an image suitability score of the classification network 30 shown in FIG. 1 . Fig. 5 (a) shows a general GAN cycle consistency loss determination method, and Fig. 5 (b) illustrates an image suitability score determination method according to the present embodiment.

Referring to Fig. 5(a), the general GAN cycle consistency loss determination method is the image X' generated by inversely transforming the fake image Y to determine how accurately the original image X was transformed into the image Y by the generating network. It is a way of seeing how equal to the original image X is. As described above, in the conventional GAN, an input is limited to one image and an output is limited to one image, but in the present embodiment, there may be M inputs and N image outputs. This offers the advantage of being able to guarantee different outputs for a limited input.

Referring to FIG. 5B , inputs matching the N inferred images X' may be M images X, a feature value Xv, and a text Xt. Inputs matching the N inferred images X' may be M images X, M feature values Xv, and M text Xt. Also, the inputs matching the N inferred images X' may be M images X and feature values Xv, M images X and text Xt, or M feature values Xv and text Xt. . The dotted line in FIG. 5B indicates the matching of the general cycle consistency loss method, and the solid line indicates the matching of the image suitability score determination method of the present embodiment.

In step 3400 , the control module 20 inputs each inferred image generated in step 3300 to the generating network 40 , thereby generating at least one image, at least one text, and at least one feature value from the output of the generating network 40 . At least two pieces of data are acquired. Next, the classification network 30 determines the suitability score of each inferred image generated in step 300 according to the similarity between at least two data obtained through the inverse transformation process and at least two data input in step 1000 . Accordingly, even in an environment in which it is difficult to obtain an input image sufficient for inference of the hidden image, it is possible to infer the hidden image with high accuracy by using the text describing the image or a value representing the characteristics of the image.

The suitability score determination process of the classification network 30 is a latent vector transformation of the generating network 40 in step 2000 for each of at least two data obtained through the inverse transformation process and at least two data input in step 1000, and correlation Since the feature inference process of is performed in the same manner, a detailed description thereof will be omitted. However, in step 3100, the generating network 40 generates inferred images through the fully accessed layer, but the classification network 30 includes at least two data obtained through an inverse transformation process through the fully accessed layer and at least two data inputted in step 1000. The features inferred for each of the two pieces of data are integrated, and a suitability score of each inferred image generated in step 300 is determined according to the degree of similarity indicated by the integrated features.

According to the present embodiment, the number of input images and output images may be arbitrarily set according to a user's decision. That is, since several hidden images can be output from one input image, a scene that is difficult to shoot with a drone or camera can be realized as a 360-degree video from an image from a specific scene in a sports game. The generation network 40 and the classification network 30 of this embodiment may support an image having a width:vertical ratio of 16:9 of a full HD size used in an actual CCTV. In the conventional GAN, the practical use of the image was reduced as only 1:1 was supported in the horizontal:vertical ratio.

FIG. 6 is an exemplary diagram in which all data input in step 1000 shown in FIG. 4 are images. For example, let's assume the following situation. The first image (51') shows a person standing in the street. The second image (52') shows that the person has just left the house. The third image 53' shows that the person is simply passing by. The fourth image (54') shows that the next destination is the house on the other side. The fifth image (55') shows the person wearing headphones and walking. The sixth image (56') shows the person taking the sunglasses out of his pocket. The seventh image (57') shows the person wearing sunglasses. The eighth image (58') shows the person walking while wearing sunglasses and headphones.

The classification network 30 and the generation network 40 are converted from the eight images (51', 52', 53', 54', 55', 56', 57', 58') input from the input module 10 . You can cut out only the necessary parts and use them for each learning. The eight cropped images 51, 52, 53 54, 55, 56, 57, and 58 may be a full HD image of 1920Х1080Х3. In the present embodiment, instead of inputting such an image, text describing a scene represented by an image may be input, or a value indicating characteristics of a certain image may be input.

FIG. 7 is an exemplary diagram of a case in which a part of data input in step 1000 shown in FIG. 4 is a feature value. Referring to FIG. 7 , the first type (T1) is a part displayed in bold, and instead of inputting an image, image characteristics such as coordinate values, movement speed values, relative position values, clothing type codes, clothing color codes, etc. The part replaced with the indicated value. The second type (T2) is a portion in which an image input exists as a light-marked portion.

For example, assume that there are no L=2 images among M=8 images. Since there is no input of the image 52', the image 52 can be replaced with a feature value consisting of 0.32 (coordinates), 0.42 (movement speed), 0.25 (relative position), 0.211 (clothing type code), and the image 55 '), the image 55 can be replaced with a feature value consisting of 0.73 (coordinate), 0.48 (movement speed), 0.05 (relative position), and 0.2225 (clothing type code). In this case, the number L of feature values substituted for the image becomes 2. The input module 10 receives ML = 8-2 images (51, 53, 54, 56, 57, 58) and L = 2 feature values (52, 55) into the classification network 30 and the generating network 40 enter each. When the size of the latent vector z is K, each of the classification network 30 and the generation network 40 is allocated M latent vectors of the size K corresponding to this input.

FIG. 8 is an exemplary diagram when a part of data input in step 1000 shown in FIG. 4 is text. Referring to FIG. 8 , the first type T1 is a portion indicated in bold, and is a portion replaced with text describing a scene of an image instead of an image input. The second type (T2) is a portion in which an image input exists as a light-marked portion.

For example, suppose there are no L=3 images among M=8 images. Since there is no input in image 52', image 52 may be replaced with the text "He was passing by the red brick house and...", and image 55 may be replaced by the text "He was passing by the red brick house and..." the camera shows his right side of his face..." can be replaced with the text "He wore the sunglass and his top left side.." as there is no input for the image (57'), so the image (57) is replaced with the text "He wore the sunglass and his top left side.." can be In this case, the number L of texts substituted for images becomes 3. The input module 10 receives ML=8-3=5 images (51, 53, 54, 56, 58) and L=3 texts (52, 55, 57) into the classification network 30 and the generating network 40 ) is entered for each. When the size of the latent vector z is K, each of the classification network 30 and the generation network 40 is allocated M latent vectors of the size K corresponding to this input.

FIG. 9 is an exemplary diagram of a case in which a part of data input in step 1000 shown in FIG. 4 is a feature value and text. Referring to FIG. 9 , the first type T1 is a portion displayed in bold, and is a portion in which values representing features of an image and text describing a scene of the image are replaced instead of inputting an image. The second type (T2) is a portion in which an image input exists as a light-marked portion.

For example, suppose that there are no L=4 images among M=8 images. Since there is no input for image 52', image 52 can be replaced with the text "He was passing by the red brick house and ...", and image (55) has " the camera shows his right side of his face..." can be replaced with the text "He wore the sunglass and his top left side.." as there is no input for the image (57'), so the image (57) is replaced with the text "He wore the sunglass and his top left side.." Since there is no input of image 53', image 53 can be replaced with a feature value consisting of "0.93 (coordinate), 0.42 (movement speed), 0.15 (relative position), 0.605 (clothing type code). In this case, the number L of data substituted for images becomes 4. The input module 10 includes ML = 8-4 images 51, 54, 56, 58 and L = 4 data table data 52, 53, 55, 57) are inputted to each of the classification network 30 and the generator network 40. When the size of the latent vector z is K, this input is applied to the classification network 30 and the generator network 40, respectively. M latent vectors of size K are allocated corresponding to .

FIG. 10 is a data processing process diagram of the generating network 40 for the example shown in FIG. 6 . In FIG. 10 , when all the data input in step 1000 shown in FIG. 4 are images, a process of reconstructing the images as output images in step 3300 is shown after collecting several images to form a stack image. One image is composed of a three-channel image of an R (red) channel, a G (green) channel, and a B (blue) channel. For example, the generating network 40 may reconstruct MХ3 pieces of 1920Х1080 full HD size channel images into NХ3 pieces of 1920Х1080 full HD size channel images.

FIG. 11 is a data processing process diagram of the generating network 40 for the example shown in FIG. 10 . FIG. 10 shows a process of reconstructing the data input in step 1000 shown in FIG. 4 into output images in step 3300 when the input data is a combination of at least one image, at least one text, and at least one feature value. has been For example, the generating network 40 may reconstruct (M-L)Х3 channel images of 1920Х1080 full HD size into NХ3 channels of 1920Х1080 full HD size channel images. In this case, there are L latent vectors of size K converted from at least one text and at least one feature value. 11 , it can be reconstructed from (M-L)Х3=(8-4)Х3=4Х3 channel images to NХ3=12Х3 channel images.

FIG. 12 is a detailed diagram of feature value processing and text processing shown in FIG. 4 . Referring to FIG. 12 , the text processing shown in FIG. 4 includes a plurality of convolution layers, a fully connected layer, a dropout layer, a platen layer, and a resafe This can be done through a reshape layer. The convolutional layer generates a three-dimensional feature map by extracting features of the text using a latent vector corresponding to the text. The fully connected layer integrates the features of the text represented by the three-dimensional feature map so that the three-dimensional feature map represents the characteristics of the association relationship between images from a textual point of view. In order to reduce the size of the 3D feature map, the dropout layer drops out values less than or equal to a specific value among the values constituting the inferred features. The platen layer transforms the three-dimensional feature map into one-dimensional feature data representing the characteristics of the relation between images from a textual point of view. The RESAFE layer converts one-dimensional feature data into K-size feature data representing features in a relation between images from a text point of view for data merging in the next step.

The feature value processing shown in FIG. 4 may be performed through a plurality of multilayer perceptron layers, platen layers, and resafe layers. The multilayer perceptron layer generates a three-dimensional feature map representing the characteristics of the association between images in the feature value point of view by inferring the features of the association relationship between the images in the feature value point of view using the latent vector corresponding to the feature value. . The platen layer converts the three-dimensional feature map into one-dimensional feature data representing the features of the correlation between images in terms of feature values. The RESAFE layer converts one-dimensional feature data into K-size feature data representing features of correlation between images in terms of feature values for data merging in the next step.

13 is a diagram illustrating an example of learning of the classification network 30 and the generating network 40 shown in FIG. 1 . 13 shows face images continuously taken while the CCTV rotates around a certain person. Some of these face images are assumed to be images captured by CCTV, and the rest are assumed to be images that cannot be captured by CCTV. Images assumed to be images captured by CCTV are input to the generating network 40 , and first inferred images are output from the generating network 40 according to the input. Images that are assumed to be images that cannot be photographed by CCTV are input to the generating network 40 , and second inferred images are output according to the input. Here, the first inferred images correspond to the fake images of the GAN and the second inferred images correspond to the inverse transformed images.

The first inferred images and the second inferred images output from the generation network 40 are input to the classification network 30 , and a suitability score of each of the first inferred images is output from the classification network 30 according to the input. . The classification network 30 is trained so that the classification network 30 outputs a value close to 1 according to the similarity between the first inferred images output from the generation network 40 and images assumed to be images that cannot be photographed by CCTV. make it If the suitability score output from the classification network 30 is 1, the first inferred images and the images assumed to be images that cannot be photographed by the CCTV match.

The generation network 40 updates the values of latent vectors representing the correlation between images in the direction in which the suitability score output from the classification network 30 is improved, and the value of latent vectors captured by CCTV using the updated latent vectors Generate images inferred from images assumed to be images. By updating the latent vectors, each latent vector can further express the relationship between images. The inference images generated in this way are again input to the classification network 30 , and the classification network 30 outputs a suitability score of each inferred image. As the process as described above is continuously repeated, the generating network 40 is able to generate inferred images that are closer to images assumed to be images that cannot be photographed by CCTV, and the classification network 30 becomes impossible to photograph. The ability to discriminate the similarity between images assumed to be images and inferred images is increasingly improved.

In the above, an example of learning the classification network 30 and the generating network 40 using only images has been described, but instead of the image, the classification network 30 and the generating network 40 are used for text or feature values as described above. can learn In particular, according to the present embodiment, since the suitability score output from the classification network 30 is fed back to the generating network 40 only when it is less than the reference score, competitive learning between the classification network 30 and the generating network 40 is required by the user. It can prevent repetition regardless of the quality of the hidden image. That is, according to the present embodiment, compared with the existing GAN, the hidden image of the quality required by the user can be efficiently provided while reducing the data processing burden on hardware such as a computer in which the present embodiment is implemented.

14 is a diagram illustrating an example of application of the hidden image inference method shown in FIG. 4 . In FIG. 14 , 12 output images when photographed over 360 degrees around the player can be inferred from one input image of a player during a soccer game. According to the present embodiment, even when there is only one input image, when text describing a scene of an image associated with the input image or a feature value of an image is input instead of an image, a hidden image associated with the input image can be inferred. there is. Furthermore, even in the absence of an input image, a hidden image may be inferred from a text or image feature value describing a certain scene.

According to this embodiment, it is possible to provide an image of an invisible object in a blind spot that cannot be photographed by an obstacle, such as a building, or by using the surrounding image, text, and image feature values. Even when there is only one face picture of the fugitive, 360-degree face images of the fugitive can be provided only by inputting a description of the fugitive face or a feature value. Even when there is only one picture of the escape route, images of the other escape routes may be provided only by the description of the escape route or input of a feature value.

On the other hand, the method for estimating the hidden image according to an embodiment of the present invention as described above can be written as a program executable by the processor of the computer, and can be implemented in a computer that records the program in a computer-readable recording medium and executes it. can The computer includes any type of computer capable of executing a program, such as a desktop computer, a notebook computer, a smart phone, and an embedded type computer. In addition, the structure of the data used in the embodiment of the present invention described above may be recorded in a computer-readable recording medium through various means. The computer-readable recording medium includes storage such as RAM, ROM, magnetic storage medium (eg, floppy disk, hard disk, etc.), and optically readable medium (eg, CD-ROM, DVD, etc.). includes media.

So far, with respect to the present invention, the preferred embodiments have been looked at. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified shape without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as included in the present invention.

10 ... input module
20 ... control module
30 ... classification networks
40 ... generative networks
50 ... storage
60 ... output module

Claims (10)

inputting at least two data of at least one image, at least one text, and at least one feature value;
generating, by the network, generating at least one image inferred from the input at least two data according to a correlation model between images built through learning;
determining, by the classification network, a suitability score of each of the generated inference images; and
and outputting each of the generated inference images as a hidden image between the images represented by the input at least two data based on the comparison result of the suitability score and the reference score of the generated inferred image, characterized in that it comprises the steps of: Hidden image inference method. The method of claim 1,
The determining of the suitability score comprises determining the suitability score of each inferred image from the input at least two pieces of data. 3. The method of claim 2,
The determining of the suitability score includes at least two data of at least one image, at least one text, and at least one feature value generated by inputting each of the generated inference images to the generation network and the input at least two A hidden image inference method, characterized in that the suitability score is determined according to the similarity of dog data. The method of claim 1,
The step of generating the inference image is
converting at least two of the at least one image, at least one text, and at least one feature value into latent vectors;
calculating a latent vector located between the transformed latent vectors by interpolating the transformed latent vectors; and
and generating at least one image inferred from the input at least two data by using the transformed latent vectors and the calculated latent vector. The method of claim 1,
The step of generating the inference image is
converting each input image into a latent vector; and
and inferring characteristics of a correlation between the images represented by the at least two input data from the viewpoint of each input image by using the transformed latent vector. The method of claim 1,
The step of generating the inference image is
converting each inputted text into a latent vector; and
and inferring characteristics of a correlation between images represented by the at least two input data from the viewpoint of each input text by using the transformed latent vector. The method of claim 1,
The step of generating the inference image is
converting each input feature value into a latent vector; and
and inferring characteristics of a correlation between the images represented by the at least two input data from the viewpoint of each input feature value by using the transformed latent vector. The method of claim 1,
The step of generating the inference image is
converting the input at least one image, at least one text, and at least one feature value into latent vectors;
inferring characteristics of a correlation between images represented by the at least two input data using the transformed latent vectors; and
Generating at least one image inferred from the input at least two data by integrating the inferred features and generating at least one image inferred from the integrated features according to the correlation model Hidden image inference method, characterized in that. A computer-readable recording medium in which a program for executing the method of any one of claims 1 to 8 on a computer is recorded. an input module for inputting at least two data of at least one image, at least one text, and at least one feature value;
a generation network that generates at least one image inferred from the input at least two data according to a correlation model between images built through learning;
a classification network that determines a suitability score of each of the generated inference images; and
and an output module for outputting each generated inference image as a hidden image between images represented by the input at least two data based on a comparison result of the suitability score and the reference score of the generated inference image A hidden image inference device.

Images