JP2018205885A - Image generation device and image generation method - Google Patents

Abstract

To provide a device and method that generates an image to which a visual experience of a user is visualized more accurately.SOLUTION: An image generation device 1 comprises: a conversion changeover section 3 converting activity data D of a brain to a brain activity feature vector Vb; a deep layer neural network for image recognition part 2 that generates an image feature vector from image data DG, generates a caption feature vector from caption data DL (DT), and generates a common feature vector Vc corresponding to the image feature vector and the caption feature vector; and a hostile neural network for image generation part 4 creating generation image data corresponding to the common feature vector Vc.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for generating an image.

  In recent years, as shown in Non-Patent Documents 1 and 2, a method of generating an image using a hostile neural network based on a caption sentence (description) has been proposed.

  Non-Patent Documents 3 and 4 propose a method of simultaneously learning a deep neural network for image recognition and a hostile neural network for image generation.

  On the other hand, in Patent Literature, for example, Patent Literature 1 discloses an image layout device using a neural network.

JP 2003-274139 A

S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative Adversarial Text to Image Synthesis”, arXiv: 1605.05396, 2016 H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, D. Metacas, “StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks”, arXiv: 1612.03242, 2016 V. Dumoulin, I. Belghazi, B. Poole, O. Mastropietro, A. Lamb, M. Arjovsky, A. Courville, “Adversarially Learned Inference”, arXiv: 1606.00704, 2016 J. Donahue, P. Krahenbuhl, and T. Darell, “Adversarial Feature Learning”, arXiv: 1605.09782, 2016

  Although the methods described in Non-Patent Documents 1 and 2 assume the input of natural language expressions (sentences), it is not always possible to use information that uniquely corresponds to the input sentences. When visual information related to brain activity is visualized as a video, there is a problem that the visual experience of the subject cannot be sufficiently reproduced.

  The present invention has been made to solve such a problem, and an object thereof is to provide an apparatus and a method for generating an image in which a visual experience of a subject is visualized more accurately.

  In order to solve the above problems, the present invention provides an image feature vector generating means for generating an image feature vector from image data, a caption feature vector generating means for generating a caption feature vector from text data corresponding to an image, and a caption feature vector. A common feature vector generating unit that generates a common feature vector according to the image feature vector, an image generating unit that generates generated image data according to the common feature vector, and a conversion that converts brain activity data into a brain activity feature vector Means for comparing the brain activity feature vector with the vector generated by the image feature vector generation unit and the common feature vector, and selectively supplying the brain activity feature vector to the channel that generates the vector having the highest correlation. An image generation apparatus including the above is provided.

  In order to solve the above problem, the present invention provides a first step of generating an image feature vector from image data, a second step of generating a caption feature vector from language data corresponding to the image data, and an image feature There is provided an image generation method including a third step of generating a common feature vector according to a vector and a caption feature vector, and a fourth step of generating generated image data according to the common feature vector.

  ADVANTAGE OF THE INVENTION According to this invention, the apparatus and method which produce | generate the image which visualized the user's visual experience more correctly can be provided.

It is a block diagram which shows the structure of the image generation apparatus 1 which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the deep-layer neural network part 2 for image recognition shown by FIG. It is a block diagram which shows the structure of the adversity neural network part 4 for image generation shown by FIG. It is a 1st flowchart which shows the image generation method which concerns on Embodiment 1 of this invention. It is a 2nd flowchart which shows the image generation method which concerns on Embodiment 1 of this invention. It is a 3rd flowchart which shows the image generation method which concerns on Embodiment 1 of this invention. It is a figure which shows the production | generation method of the caption feature vector in step S12 shown by FIG. It is a block diagram which shows the structure of the image generation apparatus 10 which concerns on Embodiment 2 of this invention. It is a block diagram which shows the structure of the deep-layer neural network part 12 for an image encoding shown by FIG. It is a block diagram which shows the structure of the deep-layer neural network part 14 for image decoding shown by FIG. It is a block diagram which shows the structure of the image generation apparatus 20 which concerns on Embodiment 3 of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts.

  An image generation apparatus and an image generation method according to an embodiment of the present invention generate an image visualizing a visual experience recognized by a user's cranial nerve activity. In generating the main image, simple information indicating visual elements is exhaustively extracted, and linguistic information related to the extracted information is supplementarily used. That is, an image recognized by the user is used as a query, and a similar image is automatically generated while using language information. Hereinafter, the image generation apparatus and the image generation method will be described in detail.

[Embodiment 1]
FIG. 1 is a block diagram showing a configuration of an image generation apparatus 1 according to Embodiment 1 of the present invention. As shown in FIG. 1, the image generation apparatus 1 includes an image recognition deep neural network unit 2, a conversion switching unit 3, an image generation hostile neural network unit 4, a random number generation unit 5, and a display unit 6. The storage unit 7 is provided.

  The image recognition deep neural network unit 2 is connected to the conversion switching unit 3, and the image generation hostile neural network unit 4 is connected to the image recognition deep neural network unit 2 and the random number generation unit 5. The display unit 6 is connected to the image generation hostile neural network unit 4 and the storage unit 7, and the storage unit 7 is further connected to the image generation hostile neural network unit 4.

  Here, the deep neural network unit 2 for image recognition generates a language / image common feature vector Vc according to the input image data DG and caption data DL (and other text data DT) corresponding to the image. Further, the conversion switching unit 3 converts the brain activity data D at the time of the input image observation into a brain activity feature vector Vb, and according to the regression result signal R supplied from the image recognition deep neural network unit 2. The channel having the highest correlation with the brain activity data D is selected, and the brain activity feature vector Vb is supplied to the channel.

  Further, the image generation hostile neural network unit 4 creates a generated image according to the image data DG, the language / image common feature vector Vc, and the random number vector Vr.

  The random number generation unit 5 generates a random number vector Vr, and the display unit 6 displays a generated image created by the image generation hostile neural network unit 4 or stored in the storage unit 7. The storage unit 7 also stores the generated image created by the image generation hostile neural network unit 4.

  FIG. 2 is a block diagram showing a configuration of the image recognition deep neural network unit 2 shown in FIG. As shown in FIG. 2, the image recognition deep neural network unit 2 includes an image recognition deep neural network 21, a caption feature vector generation unit 22, and a common feature vector generation unit 23.

  The image recognition deep neural network 21 is connected to the conversion switching unit 3, and the common feature vector generation unit 23 is connected to the image recognition deep neural network 21, the caption feature vector generation unit 22, and the conversion switching unit 3.

  Here, the deep neural network for image recognition 21 is a neural network including n layers Lv1 to Lvn including a convolution operation layer and a fully connected layer, and generates an image feature vector Vg. The caption feature vector generation unit 22 generates a caption feature vector Vs from the caption data DL. The common feature vector generating unit 23 performs machine learning in advance so that the vector space in which the caption feature quantity corresponding to the image is represented matches the vector space in which the image feature quantity is represented, so that the language / image common feature is obtained. The vector Vc can be generated.

  Then, the conversion switching unit 3 shown in FIG. 1 applies outputs such as regression and canonical correlation analysis, outputs from the layers Lv1 to Lvn, and image features generated by the deep neural network 21 for image recognition. According to the regression result signal R indicating the regression results respectively calculated between the vector Vg and the language / image common feature vector Vc generated by the common feature vector generation unit 23 and the brain activity feature vector Vb, the regression result Is selected, that is, the channel that generates the vector having the highest correlation with the brain activity data D, and the brain activity feature vector Vb is directly supplied to the channel during the execution phase described later. Allows generation of a common feature vector Vc.

  Therefore, for example, when the correlation between the brain activity data D and the output from the layer Lv2 is the highest, the conversion switching unit 3 uses the layer Lv2 as the optimum channel, and the brain activity data D and the language / image common feature vector When the degree of correlation with Vc is the highest, the conversion switching unit 3 uses the common feature vector generation unit 23 as the optimum channel and supplies the brain activity feature vector Vb.

  FIG. 3 is a block diagram showing a configuration of the image generation hostile neural network unit 4 shown in FIG. As shown in FIG. 3, the image generation hostile neural network unit 4 includes an image generation deep neural network 41, a discrimination deep neural network 42, and an error detector 43.

  The image generation deep neural network 41 is connected to the common feature vector generation unit 23, the random number generation unit 5, and the error detector 43. Further, the discrimination deep neural network 42 is connected to the common feature vector generation unit 23, the image generation deep neural network 41, and the error detector 43.

  Here, the image generation deep neural network 41 is a neural network including n ′ layers Lg1 to Lgn ′, and each layer includes a convolution operation layer and a fully connected layer. In the deep neural network 41 for image generation, an image is synthesized based on the random number vector Vr while using information of the language / image common feature vector Vc.

  On the other hand, the distinguishing deep neural network 42 is a neural network including n ″ layers Ld1 to Ldn ″, and each layer includes a convolution operation layer and a fully connected layer. In the discrimination deep neural network 42, image data (actual image) DG in an image database or the like and a generation image created by the image generation deep neural network 41 are input, and generation is performed to determine whether the input image is a real image. A discrimination signal is output depending on whether the image is an image.

  Further, the error detector 43 outputs a determination signal indicating the generated image even though the deep neural network for determination 42 inputs the actual image, or outputs a determination signal indicating the actual image even though the generated image is input. In this case, it is considered that an erroneous discrimination signal is generated by the discrimination deep neural network 42, and the weighting coefficient used in the calculation is corrected for the image generation deep neural network 41 and the discrimination deep neural network 42. Supply signal err.

  In this way, in the image generation hostile neural network unit 4, the image generation deep neural network 41 learns to output a generated image that cannot be distinguished from the actual image for the determination deep neural network 42, and performs the determination. The deep neural network 42 learns so that a real image and a generated image can be correctly distinguished.

  4 to 6 are flowcharts showing an image generation method according to Embodiment 1 of the present invention. As will be described in detail later, FIG. 4 shows a first learning phase in which optimization learning of the image recognition deep neural network unit 2 is performed from image data DG and image caption data DL, and FIG. 5 shows image data DG. And a second learning phase in which the optimization learning of the conversion switching unit 3 is performed using the brain activity data D. FIG. 6 shows an execution phase using only brain activity data D.

  In the following, a case where the image generation method shown in FIGS. 4 to 6 is executed by using the image generation apparatus 1 shown in FIG. 1 will be described as an example. Needless to say, the present invention is not limited to the case of using the generation apparatus 1 and may be executed using other means.

In the first step S11 shown in FIG. 4, the image recognition deep neural network 21 generates an image feature vector Vg based on the image data DG recognized by the brain. The image feature vector Vg is generated by using, for example, VGG19 (“Very Deep Convolutional Networks for Large-Scale Image Recognition”, K. Simonyan, A. Zisserman, arXiv: 1409.1556) often used in computer vision research. The

  In the next step S12, the caption feature vector generation unit 22 generates a caption feature vector Vs from the caption data DL corresponding to the image. Hereinafter, a method for generating the caption feature vector Vs will be described in detail with reference to FIG.

  FIG. 7 shows a case where the caption feature vector Vs is generated based on the caption data DL corresponding to the image data DG.

As shown in FIG. 7, first, the caption data DL corresponding to the image is collated with a word list W prepared in advance, and the listed words w _{1 to} w _k are extracted to count the number of appearances. The list word appearance frequency vector w_o is generated.

  On the other hand, text data DT not related to an image is also collated with the word list W to generate a list word feature matrix A that is a set of feature vectors of all list words. The list word feature matrix A is generated by natural language processing such as WordTubec (Word2vec), which is a technique for expressing each word as a vector by machine learning of the appearance relation of each word from text data. Based on the generated list word feature matrix A, a list word similarity matrix C is calculated.

  Next, the list word appearance weight vector w_o ′ is generated by multiplying the list word appearance frequency vector by the list word similarity matrix C.

  Here, multiplying the list word appearance frequency vector by the list word similarity matrix C takes into account the contribution of words that do not appear directly in the caption using the similarity between languages with other texts. Means.

  Next, a caption feature vector Vs is generated by multiplying the list word appearance weight vector w_o ′ by the list word feature matrix A.

  Thereafter, in step S13, the common feature vector generation unit 23 generates the brain activity feature vector Vb generated by the conversion switching unit 3, the image feature vector Vg generated by the image recognition deep neural network 21, and the caption feature vector generation unit 22. The language / image common feature vector Vc is generated according to the caption feature vector Vs generated by the above.

  Here, the common feature vector generating unit 23 is a well-known paper widely used in natural language processing (“Unifying visual semantic embedings with multimodal neural language models”, Kiros, Salakhutdinov, Zemel, arXiv) in addition to regression analysis and canonical correlation analysis. : 1411.2539) is learned in advance so that the image feature vector and the caption feature vector are converted into a matching vector.

  Next, in step S14, the image generation hostile neural network unit 4 creates generated image data according to the language / image common feature vector Vc and the random number vector Vr. The created generated image data is displayed on the display unit 6 or stored in the storage unit 7 in accordance with a user operation.

  In the second learning phase, in step S21 shown in FIG. 5, brain activity data D composed of data measured using a brain / nerve activity measuring device for each region (part) of the brain is acquired. .

  Next, in step S22, the conversion switching unit 3 converts the brain activity data D to generate a brain activity feature vector Vb.

  Next, in step S23, the conversion switching unit 3 compares the output vector of each layer of the image recognition deep neural network 21, the common feature vector Vc, and the brain activity feature vector Vb, and responds to the conversion with the best regression result. Optimization learning and a channel that generates a vector having the highest correlation with the brain activity feature vector Vb is selected as the optimum channel.

  In the execution phase, the brain activity data D is acquired in step S31 shown in FIG. 6, the conversion switching unit 3 converts the brain activity data D into the brain activity feature vector Vb in step S32, and then the conversion is switched in step S33. The unit 3 supplies the brain activity feature vector Vb to the optimum channel of the deep neural network unit 2 for image recognition.

  In step S34, the image generation hostile neural network unit 4 creates generated image data according to the language / image common feature vector Vc generated by the image recognition deep neural network unit 2 based on the brain activity feature vector Vb. To do.

  As described above, according to the image generation device 1 according to the first embodiment of the present invention, an image is generated using not only visual information but also feature amount expression of language information, so that the visual experience of the user can be visualized more accurately. Images can be created.

[Embodiment 2]
In the following, only differences from the first embodiment will be described, and description of common points will be omitted.

  FIG. 8 is a block diagram showing the configuration of the image generation apparatus 10 according to Embodiment 2 of the present invention. As shown in FIG. 8, the image generation device 10 according to the second embodiment has the same configuration as the image generation device 1 according to the first embodiment, but the image recognition deep neural network unit 2 and the image generation hostile. Instead of the neural network unit 4, language data DL such as caption data corresponding to an image is supplied, a caption feature vector generating unit 22 for generating a caption feature vector Vs, an image encoding deep neural network unit 12, and an image An image decoding deep neural network unit 14 to which data DG and caption feature vector Vs are supplied is provided.

  The conversion switching unit 13 shown in FIG. 8 has the same function as the conversion switching unit 3 shown in FIG. 1, but the output from each layer Le1 to Len and the estimated random number generated by the vector generation unit 16 According to the regression result signal R indicating the regression result calculated between the vector EVr, the estimated caption feature vector EVs, and the brain activity feature vector Vb, the vector having the best regression result, that is, the highest correlation degree Is selected, and the brain activity feature vector Vb is supplied to the selected channel during the execution phase.

  FIG. 9 is a block diagram showing a configuration of the image coding deep neural network unit 12 shown in FIG. As shown in FIG. 9, the image coding deep neural network unit 12 includes an image coding deep neural network 15 and a vector generation unit 16.

  The image encoding deep neural network 15 is connected to the conversion switching unit 13 and the error detector 17, and the vector generation unit 16 is connected to the image encoding deep neural network 15 and the conversion switching unit 13.

  Here, the image encoding deep neural network 15 is a neural network including n layers Le1 to Len including a convolution operation layer and a fully connected layer, and generates an image feature vector Vg. Further, the vector generation unit 16 receives the image feature vector Vg and outputs an estimated caption feature vector EVs and an estimated random number vector EVr.

  FIG. 10 is a block diagram showing a configuration of the image decoding deep neural network unit 14 shown in FIG. As shown in FIG. 10, the image decoding deep neural network unit 14 has the same configuration as the image generation hostile neural network unit 4 shown in FIG. The random number vector Vr and the caption feature vector Vs generated by the caption feature vector generation unit 22 are input to generate an image.

  In addition, the discrimination deep neural network 19 includes a caption feature vector Vs, a random number vector Vr, and a generated image (hereinafter referred to as “generated image related data”), or an estimated random number vector EVr and an estimation generated by the vector generation unit 16. Caption feature vector EVs and image data DG (hereinafter referred to as “real image related data”) are input, and it is determined whether the input data is generated image related data or real image related data.

  Here, as described above, the implementation of the image generation device 10 using the estimated caption feature vector EVs and the estimated random number vector EVr is disclosed in a known paper (“Adversarial Feature Learning”, Jeff Donahue, Philipp Krahenbuhl, Trevor Darrell, arXiv: 1605.09782 "Adversarially Learned Inference", Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, Aaron Courville, arXiv: 1606.00704).

  According to the image generation apparatus 10 according to the second embodiment of the present invention as described above, an image is generated with reference to not only the image data DG but also language data such as caption data DL corresponding to the image. In an environment where language data can be used, an image closer to the real image recognized by the brain can be created.

[Embodiment 3]
FIG. 11 is a block diagram showing a configuration of the image generation apparatus 20 according to Embodiment 3 of the present invention. As shown in FIG. 11, the image generation apparatus 20 has a configuration in which the image recognition deep neural network unit 2 according to the first embodiment is further added to the image generation apparatus 10 according to the second embodiment. However, the image decoding deep neural network unit 14 uses the language / image common feature vector Vc generated in the image recognition deep neural network unit 2 instead of the caption feature vector Vs generated by the caption feature vector generation unit 22. Is entered.

  According to the image generation apparatus 20 according to the third embodiment of the present invention having such a configuration, the image generation apparatus 1 according to the first embodiment and the image generation apparatus according to the second embodiment are combined in one apparatus. Both functions can be realized.

1, 10, 20 Image generation device 2 Image recognition deep neural network unit 3, 13 Conversion switching unit 4 Image generation hostile neural network unit 12 Image encoding deep neural network unit 14 Image decoding deep neural network unit 15 Image Deep neural network for encoding 16 Vector generating unit 18 Deep neural network for image decoding 21 Deep neural network for image recognition 22 Caption feature vector generating unit 23 Common feature vector generating unit 41 Image generating deep neural network 19, 42 Discriminating deep layer Neural network 17, 43 Error detector

Claims (8)

Image feature vector generation means for generating an image feature vector from image data;
Caption feature vector generating means for generating a caption feature vector from text data corresponding to an image;
Common feature vector generation means for generating a common feature vector according to the caption feature vector and the image feature vector;
Image generating means for generating generated image data according to the common feature vector;
Conversion means for converting brain activity data into brain activity feature vectors;
A selection unit that compares the brain activity feature vector with the vector generated by the image feature vector generation unit and the common feature vector, and selectively supplies the brain activity feature vector to a channel that generates a vector having the highest correlation. An image generation apparatus comprising: The image feature vector generating means is constituted by a deep neural network,
The image generation apparatus according to claim 1, wherein the image generation unit is configured by a hostile neural network.   The image generation apparatus according to claim 1, wherein the caption feature vector generation unit generates the caption feature vector in consideration of similarity between languages.   The image generation apparatus according to claim 1, wherein the image generation unit further generates the generated image data in accordance with the caption feature vector. The image feature vector generating means is constituted by a deep neural network,
The image generation device according to claim 4, wherein the image generation means is configured by an adversarial neural network. A first step of generating an image feature vector from the image data;
A second step of generating a caption feature vector from language data corresponding to the image data;
A third step of generating a common feature vector according to the image feature vector and the caption feature vector;
And a fourth step of generating generated image data according to the common feature vector.   The image generation method according to claim 6, wherein in the second step, the caption feature vector is generated in consideration of similarity between words. The image generation method according to claim 6, wherein in the fourth step, the generated image data is further created based on the image data and language data.

Images